Railway container corner parts abnormal state detection method and related equipment

Abstract

The present invention relates to the field of image processing technology, and discloses a method for detecting abnormal state of corner fittings of railway containers and related equipment. The method comprises: obtaining a set of railway freight empty-car passing images, extracting the F‑TR lock image in each empty-car passing image based on an image segmentation algorithm; obtaining a set of railway freight loaded-box passing images, superimposing each F‑TR lock image on the preset corner fitting position of each loaded-box passing image in the loaded-box passing image set, and obtaining a training data set; performing model training based on the training data set to obtain a target detection model with the F‑TR lock as the detection target; performing F‑TR lock detection on the loaded-box passing image to be identified based on the target detection model, and determining the abnormal state of the loading corner fitting according to the detection result. The present invention can construct a data set when the samples are sparse, improve the prediction quality of the machine learning model, and thus effectively realize the recognition and alarm of abnormal state of container corner fittings.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for detecting abnormal state of corner fittings of railway containers and related equipment. Background Art

[0002] Abnormal corner fittings of railway containers are a major problem in the cargo inspection process. If the corner fittings are not completely seated in the F-TR lock head, the container may overturn during transportation, causing significant losses to the safety of people and property along the line.

[0003] The abnormal state detection technology of railway container corner fittings can assist operators to timely discover and deal with safety risks. In this scenario, artificial intelligence methods such as R-CNN, SSD or YOLO are usually used for F-TR lock feature detection. By identifying the features of the F-TR lock, it is determined whether there are corner fittings that are completely out of place. However, corner fittings that are completely out of place are the key manual inspection objects of cargo inspection operations at each station. The problem has not been dealt with and there are few sample images available for collection, so it is impossible to obtain an accurate and reliable target detection model, resulting in the inability to effectively realize the abnormal state detection of railway container corner fittings. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for detecting abnormal state of corner fittings of railway containers and related equipment in view of the problems existing in the prior art.

[0005] In a first aspect, the present invention provides a method for detecting abnormal state of a corner fitting of a railway container, comprising:

[0006] Obtain a set of railway freight empty train images, and extract the F-TR lock image from each empty train image in the set based on the image segmentation algorithm;

[0007] A railway freight container passing image set is obtained, and each F-TR lock image is superimposed on the preset corner position of each container passing image in the container passing image set based on target detection and gradient threshold fusion to obtain a training data set;

[0008] Model training is performed based on the training data set to obtain a target detection model with the F-TR lock as the detection target;

[0009] Based on the target detection model, F-TR lock detection is performed on the image of the container passing through the vehicle to be identified, and the abnormal status of the container corner fittings is determined based on the detection results.

[0010] In a second aspect, the present invention provides a railway container corner fitting abnormal state detection system, comprising:

[0011] The F-TR lock image acquisition module is used to acquire a set of railway freight empty train images, and extract the F-TR lock image in each empty train image in the empty train image set based on an image segmentation algorithm;

[0012] A training data synthesis module is used to obtain a railway freight container passing image set, and to superimpose each F-TR lock image on a preset corner position of each container passing image in the container passing image set based on target detection and gradient threshold fusion to obtain a training data set;

[0013] A model training module is used to perform model training based on a training data set to obtain a target detection model with the F-TR lock as the detection target;

[0014] The state detection module is used to perform F-TR lock detection on the image of the container passing through the vehicle to be identified based on the target detection model, and determine the abnormal state of the container corner fittings according to the detection results.

[0015] In a third aspect, the present invention provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the method for detecting abnormal status of corner fittings of railway containers as provided in the above technical solution.

[0016] In a fourth aspect, the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for detecting abnormal state of corner fittings of railway containers as provided in the above technical solution is implemented.

[0017] In a fifth aspect, the present application further provides a computer program product, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the abnormal state detection method for railway container corner fittings provided by the above solution.

[0018] The beneficial effects of the present invention are as follows: the present invention is based on image feature adaptation of target detection and gradient threshold fusion, and superimposes the F-TR lock image in the empty vehicle passing image to the container corner fitting position in the loaded vehicle passing image, thereby realizing the synthesis of training data, thereby being able to construct a data set when the samples are sparse, improving the prediction quality of the machine learning model, and being able to solve the needs of identifying and warning of abnormal states of container corner fittings; and the present invention has little impact on the detection speed of the production environment, meeting the needs of the cargo inspection site to complete operations in a shorter time.

[0019] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a method for detecting abnormal state of a corner fitting of a railway container, shown as an exemplary embodiment of the present application;

[0021] Figure 2 for Figure 1 A flowchart of step S2 in the illustrated embodiment in an exemplary embodiment;

[0022] Figure 3 for Figure 1 A flowchart of step S2 in the illustrated embodiment in another exemplary embodiment;

[0023] Figure 4 for Figure 2 or Figure 3 A flowchart of step S24 in the illustrated embodiment in an exemplary embodiment;

[0024] Figure 5 for Figure 1 A flowchart of step S4 in the illustrated embodiment in an exemplary embodiment;

[0025] Figure 6 A flow chart of a method for detecting abnormal state of a corner fitting of a railway container shown as another exemplary embodiment of the present application;

[0026] Figure 7 This is an image of an empty vehicle passing by an exemplary embodiment of the present application;

[0027] Figure 8 For Figure 7 The small area image containing F-TR lock features extracted from the empty vehicle passing image shown;

[0028] Fig. 9 for Figure 8 The image after the F-TR lock marks the polygon outline;

[0029] Fig.10 This is an image of a container passing a vehicle shown in an exemplary embodiment of the present application;

[0030] Fig.11 For identification Fig.10 Image after the center corner piece is in place;

[0031] Fig.12 for Fig. 9 and Fig.11 A composite image obtained after image fusion;

[0032] Fig.13 This is a schematic diagram of F-TR lock feature detection results shown in an exemplary embodiment of the present application;

[0033] Fig.14A block diagram of a railway container corner fitting abnormal state detection system shown as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0035] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0036] The embodiment of the present invention proposes a target detection method for abnormal status of railway container corner fittings, which integrates feature adaptation, data enhancement, machine learning and other methods to detect the features of F-TR locks on container images.

[0037] The solution provided in the embodiments of the present invention can be executed by any electronic device, for example, a terminal device, or jointly executed by a terminal device and a server. The server can be an independent server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and is not limited here.

[0038] Figure 1 The flowchart of the method for detecting abnormal state of corner fittings of railway containers is shown as an exemplary embodiment of the present application. Figure 1As shown, in an exemplary embodiment, the abnormal state detection of the railway container corner fitting may include steps S1 to S4, which are described in detail as follows:

[0039] S1, obtain a set of railway freight empty train images, and extract the F-TR lock image in each empty train image in the set based on the image segmentation algorithm.

[0040] It should be noted that the empty vehicle passing image set includes a large number of empty vehicle passing images, and the empty vehicle passing images are image data obtained when the vehicle is in the state of an empty flat car and an empty container dedicated flat car.

[0041] S2, obtain a set of railway freight box passing images, and based on target detection and gradient threshold fusion, superimpose each F-TR lock image on the preset corner position of each box passing image in the box passing image set to obtain a training data set.

[0042] It should be noted that the container-carrying vehicle image set includes a large number of container-carrying vehicle images, and the container-carrying vehicle images are image data acquired when the vehicle is in a flat car state with containers.

[0043] The embodiment of the present invention performs image fusion of the F-TR lock image and the image of the vehicle with the box passing by based on target detection and gradient threshold fusion, thereby obtaining training data.

[0044] S3, performing model training based on the training data set to obtain a target detection model with the F-TR lock as the detection target.

[0045] S4, based on the target detection model, F-TR lock detection is performed on the image of the container passing through the vehicle to be identified, and the abnormal state of the container corner fittings is determined according to the detection results.

[0046] The embodiment of the present invention is based on image feature adaptation based on target detection and gradient threshold fusion, and superimposes the F-TR lock image in the empty vehicle passing image to the container corner fitting position in the loaded vehicle passing image to achieve training data synthesis, thereby being able to construct a data set when samples are sparse, improve the prediction quality of the machine learning model, and solve the needs of identifying and warning of abnormal conditions of container corner fittings.

[0047] The embodiment of the present invention facilitates the difference comparison of rare images with complex image features at multiple measuring points, and can quickly and accurately identify abnormal states of container corner fittings without adding additional neural network layers. It maintains invariance to container position, loading method and vehicle type, and maintains good stability to changes in viewing angle and image noise. In addition, the present invention has little impact on the detection speed of the production environment, meeting the need to complete operations in a shorter time at the cargo inspection site.

[0048] Optionally, Figure 1In an exemplary embodiment, step S1 in the illustrated embodiment may include steps S11 to S15. The F-TR lock image is obtained through the above steps, which are described in detail as follows:

[0049] S11: Obtain an image set of empty railway freight cars. Specifically, a linear array camera is used to obtain images of railway freight cars without containers (i.e., empty railway freight cars), and a large number of empty flat cars and empty container dedicated flat cars are collected.

[0050] S12: Cut a local image of the F-TR lock position from each empty train image in the empty train image set. Specifically, take the loading state images of the railway trains without containers on the left and right sides, and for each side image I, take the coordinates of the upper left corner of the image as point (0,0), and take three rectangular areas:

[0051]

[0052] The above rectangular part is extracted to obtain a small area local image containing the F-TR lock feature.

[0053] S13: Extract the F-TR lock polygonal image from the local image. Specifically, use image editing software to process the local image, mark along the contour features of the F-TR lock, and obtain the polygonal contour marking data of each F-TR lock.

[0054] S14: Using the local image to train the instance segmentation model. Specifically, based on the YOLOv8-seg model, using the polygonal contour annotation data of the F-TR lock, and based on the local image as the background image, the instance segmentation model is trained for the F-TR lock feature extraction on the empty flat car image.

[0055] S15: Use the instance segmentation model to cut all the empty vehicle passing images to obtain the F-TR lock image.

[0056] In the embodiment of the present invention, the F-TR lock image in each empty vehicle passing image is extracted based on the image segmentation algorithm, which can automatically and accurately locate and segment the F-TR lock area, avoiding the tedious process of manually labeling all the original image data, and greatly improving the data processing efficiency; at the same time, since local images are used for model training, background interference and data redundancy are reduced, providing a reliable data basis for the subsequent container corner fitting abnormal state detection process.

[0057] See also Figure 2 , Figure 2 for Figure 1 The step S2 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 2As shown, step S2 may specifically include steps S21 to S25, and the training data synthesis based on target detection and gradient domain fusion is realized through the above steps, which is described in detail as follows:

[0058] S21, based on the target detection algorithm, determine the preset corner fitting position of the container in each container passing image in the container passing image set.

[0059] In the embodiment of the present invention, a loading state image of a railway freight car loaded with containers (i.e., a container-carrying freight car image) is acquired by a line array camera, and a large number of flat car image data set samples loaded with containers are collected.

[0060] The embodiment of the present invention detects the positions of all corner fittings in the current container passing image based on the container corner fitting target detection model. The container corner fitting identification frame on a single side passing image should present an obvious double-layer distribution pattern, with 4 (double-box side-by-side loading) or 2 (single-box loading) corner fitting identification frames on the upper and lower sides. In the embodiment of the present invention, the lower layer identification frame is taken as the preset corner fitting position. That is, the embodiment of the present invention records the position of the lower layer identification frame in the corner fitting identification frame presented in the double-layer distribution pattern in the detection result as the preset corner fitting position. That is, the bottom corner fitting of the container is taken as the preset corner fitting, and the position where the bottom corner fitting is located is taken as the preset corner fitting position.

[0061] S22, aligning the lower left corner of the F-TR lock image to be superimposed with the lower left corner of the preset corner piece position in the container passing vehicle image to be superimposed, to obtain a corresponding superimposed layer.

[0062] S23, reconstructing the superimposed layers based on a gradient domain fusion algorithm to obtain a corresponding composite image.

[0063] It should be noted that directly superimposing layers will result in unnatural transitions around the F-TR lock image, and the model may therefore learn incorrect features. Therefore, the embodiment of the present invention uses a gradient domain fusion method to eliminate unnatural transitions and enhance the model recognition effect.

[0064] S24, extracting the container image in the synthesized image based on the target detection algorithm, and determining the relative coordinates of the F-TR lock image on the container image.

[0065] S25, all the container images obtained constitute a training data set.

[0066] In the embodiment of the present invention, the training data set is generated by layer alignment and gradient domain fusion algorithm, which can ensure the authenticity and naturalness of the synthesized image and effectively avoid edge breaks and brightness jumps caused by abrupt splicing. This method significantly expands the number of training samples, improves the diversity of the data set, and enables the model to learn more robust feature expressions, thereby improving the generalization ability and practical application effect of the detection model.

[0067] See also Figure 3 , Figure 3 for Figure 1 The step S2 in the embodiment shown is a flow chart in another exemplary embodiment. Figure 3 As shown, step S2 may specifically include steps S201 to S25, and the training data synthesis based on target detection and gradient domain fusion is realized through the above steps, which is described in detail as follows:

[0068] S201, using a normalized brightness calculation method to determine a first brightness value of each image of a box-carrying vehicle passing by, and adjusting the first brightness value until the difference with a preset brightness reference value is less than a preset threshold value.

[0069] It should be noted that the lighting of cargo inspection images usually has a phenomenon of slightly overexposure in the sky and slightly underexposure in the carriage. In order to avoid the inability to correctly select the exposure value when the on-site camera collects images, which affects the recognition effect in the case of sparse samples, the following formula is proposed to calculate the overall brightness of the image after suppressing the highlight part, and logistic regression is used to normalize the results:

[0070]

[0071]

[0072] In the formula, is the final normalized brightness value, ranging from [0, 1], is the brightness value of pixel i, is the weight of pixel i, T is the brightness threshold, S is the smoothing factor, k is the steepness parameter of the logistic function, μ is the center point of the logistic function, is the normalized local contrast of the effective area, is the normalized local entropy of the effective area, and p is the influencing factor of contrast and entropy.

[0073] The above formula is based on the brightness value of each pixel , brightness threshold T and smoothing factor S are used to calculate the weight of each pixel to suppress the influence of the highlight area, and by introducing local contrast and entropy, the formula can consider the local characteristics of the image when calculating the final brightness value.

[0074] In the embodiment of the present invention, 0.75 is given as the image brightness reference value, and the image whose brightness calculation result deviates too much from the reference value is brightened / darkened until it approaches the brightness reference value. In the embodiment of the present invention, the preset threshold value can be 0.1.

[0075] S202, using a normalized brightness calculation method to determine a second brightness value of the F-TR lock image to be superimposed, and adjusting the second brightness value until the difference between the second brightness value and the first brightness value of the container passing image to be superimposed is less than a preset threshold.

[0076] Specifically, the F-TR lock image extracted from S1 is randomly selected, and the first brightness value of the image of the loaded box passing the vehicle to be stacked and the second brightness value of the F-TR lock image to be stacked are calculated using the normalized brightness calculation formula, and the second brightness value of the F-TR lock image is adjusted until the difference between the second brightness value and the first brightness value is less than the preset threshold, that is, the difference .

[0077] S21, based on the target detection algorithm, determine the preset corner fitting position of the container in each container passing image in the container passing image set.

[0078] The embodiment of the present invention uses a linear array camera to obtain the loading state image of a railway freight car loaded with containers (i.e., the image of the container passing the train), and collects a large number of flat car image data set samples loaded with containers. The container corner fittings target detection model is used to detect the positions of all corner fittings of the container in the image of the container passing the train. The container corner fittings recognition frame on a single side passing image should show an obvious double-layer distribution pattern, with 4 (double-box side-by-side loading) or 2 (single-box loading) corner fittings recognition frames on the upper and lower sides. In the embodiment of the present invention, the lower layer recognition frame is removed as the preset corner fitting position.

[0079] S22, aligning the lower left corner of the F-TR lock image to be superimposed with the lower left corner of the preset corner piece position in the container passing vehicle image to be superimposed, to obtain a corresponding superimposed layer.

[0080] S23, reconstructing the superimposed layers based on a gradient domain fusion algorithm to obtain a corresponding composite image.

[0081] It should be noted that directly superimposing layers will result in unnatural transitions around the F-TR lock image, and the model may therefore learn incorrect features. Therefore, the embodiment of the present invention uses a gradient domain fusion method to eliminate unnatural transitions and enhance the model recognition effect.

[0082] S24, extracting the container image in the synthesized image based on the target detection algorithm, and determining the relative coordinates of the F-TR lock image on the container image.

[0083] S25, all the container images obtained constitute a training data set.

[0084] It should be noted that S202 may also be executed after S21 and before S22, and the specific execution order is not limited in the embodiment of the present invention.

[0085] In the embodiment of the present invention, the gradient domain fusion method and image feature adaptation are used to achieve target detection, which can construct a data set when the samples are sparse, and reduce the negative impact of lighting conditions and camera parameters during the inference process. The embodiment of the present invention is convenient for performing difference comparison on rare images with complex image features at multiple measurement points, and can quickly and accurately identify abnormal states of container corner fittings without adding additional neural network layers. It maintains invariance to the container position, loading method and vehicle type, and maintains good stability to changes in viewing angle and image noise.

[0086] See also Figure 4 , Figure 4 for Figure 2 The step S24 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 4 As shown, step S24 may specifically include steps S241 to S244. The container image is obtained through the above steps, which is described in detail as follows:

[0087] S241, predicting the edge position of the container on the synthetic image using the container edge target detection model to obtain an edge position prediction result;

[0088] S242, using the F-TR locked image coordinates in the overlay layer to perform auxiliary calibration on the edge position prediction result;

[0089] S243, if the edge position prediction result cannot completely cover the X-axis coordinates of the left and right sides of the F-TR lock image, the boundary coordinate values ​​of the container edge position prediction box are expanded until they can be completely covered;

[0090] S244, taking the image within the container edge position prediction box as the container image.

[0091] After obtaining the container image, the F-TR lock coordinates after cutting the container edge image need to be calculated. For each F-TR lock coordinate frame, the left boundary coordinate value of the container edge position prediction frame on the original image (i.e., the empty vehicle image) is obtained, which is recorded as ; Subtract the left boundary coordinate value from the horizontal coordinates of each vertex in the current F-TR lock coordinate frame , the relative horizontal coordinates of the F-TR lock coordinate frame in the cropped container edge image can be obtained.

[0092] In the embodiment of the present invention, preliminary positioning is performed through the container edge target detection model, and auxiliary calibration and adaptive boundary expansion are performed in combination with the F-TR lock image coordinates, and finally the required container edge image is extracted, which can ensure the integrity and accuracy of the target area and effectively avoid the truncation and loss of target features. This method significantly improves the quality and reliability of training data through a multi-step optimization positioning process, which not only reduces the interference of the background area, but also ensures the complete retention of the target features, which helps the model establish a more accurate feature expression, thereby improving the detection performance.

[0093] See also Figure 5 , Figure 5 for Figure 1 The step S4 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 5 As shown, step S4 may specifically include steps S41 to S43, and the F-TR lock detection is performed through the above steps, and the abnormal state of the packing corner piece is determined according to the detection result, which is described in detail as follows:

[0094] S41, obtaining an image of a vehicle with containers to be identified, and extracting an edge portion of the lower half of the container image in the image of the vehicle with containers to be identified.

[0095] In the embodiment of the present invention, the images of vehicles passing by that need to be identified are collected by shooting with a three-way linear array camera, and the vehicle type classification model is used to screen out the images of vehicles passing by that are flat cars loaded with containers, and prepare for identification.

[0096] The container edge target detection model is used to detect the screened vehicle images to obtain the container edge area image, and its lower half is captured as the final container edge image.

[0097] S42, using a normalized brightness calculation method to determine a third brightness value of the edge portion of the image, and adjusting the third brightness value until the difference between the third brightness value and the preset brightness reference value is less than a preset threshold, that is, the difference .

[0098] S43, using the target detection model shown, to perform F-TR lock detection on the edge portion image after brightness adjustment, and determine the abnormal state of the packing corner piece according to the detection result.

[0099] In the embodiment of the present invention, the image of the container flat car is accurately screened through the vehicle type classification model and the lower half of the container edge area is extracted. The normalized brightness calculation method is combined to perform dynamic brightness calibration, which can effectively eliminate the influence of illumination changes on the detection results and ensure the consistency of image data. This method establishes a complete processing flow from image acquisition to feature detection, significantly improving the accuracy and reliability of F-TR lock detection and abnormal state judgment, and making the system more environmentally adaptable and stable.

[0100] The embodiment of the present invention is a target detection method that combines traditional image processing with deep learning. From the overall structure, the entire target detection method is mainly divided into three stages: the F-TR lock feature extraction stage based on the image segmentation algorithm, the training data synthesis stage based on target detection and gradient domain fusion, and the data inference stage based on image feature adaptation. For ease of explanation, the whole process is demonstrated based on the on-site image. The detailed process is as follows Figure 6 As shown:

[0101] 1. The F-TR lock feature extraction stage based on the image segmentation algorithm mainly includes the following steps:

[0102] Step 1: Obtain high-definition vehicle images. Use a linear array camera to collect a large number of empty flat cars and empty container dedicated flat cars dataset samples. The collected image samples are as follows: Figure 7 shown.

[0103] Step 2: Cut the F-TR lock position part from the image. Take the left and right high-definition images of the vehicle passing by. For each side image I, take the coordinates of the upper left corner of the image as (0,0) and take three rectangular areas:

[0104]

[0105] Extract the above rectangular part to obtain a small area image containing the F-TR lock feature, such as Figure 8 shown.

[0106] Step 3: Extract the polygonal image of the F-TR lock. Use image editing software to select some small area images in Step 2 for processing, mark them along the contour features of the F-TR lock, and obtain the polygonal contour marking data of each F-TR lock, such as Fig. 9 As shown:

[0107] Step 4: Train the instance segmentation model. Based on the YOLOv8-seg model, using the contour features in Step 3, and based on the small area image after the rectangle is extracted as the background image, the instance segmentation model is trained for F-TR lock feature extraction on the empty flat car image. The learning rate is set to 0.005 during training, and the bounding box regression loss weight is set to 0.1.

[0108] Step 5: Predict samples on all empty flat car images. Use the instance segmentation model trained in Step 4 to identify the contours of corner parts in the remaining sample images, set the confidence level to 0.75, and use it as a feature data set.

[0109] 2. The training data synthesis stage based on target detection and gradient domain fusion mainly includes the following steps:

[0110] Step 1: Obtain high-definition images of vehicles passing by. Use a linear array camera to collect a large number of image samples of flat cars carrying containers. The collected image samples are as follows: Fig.10 shown.

[0111] Step 2: Adjust the image exposure compensation. The lighting of cargo inspection images usually has the phenomenon of slightly overexposing the sky and slightly underexposing the carriage. In order to avoid the inability to correctly select the exposure value when the on-site camera collects images, which affects the recognition effect in the case of sparse samples, the following formula is proposed to calculate the overall brightness of the image after suppressing the highlight part, and logistic regression is used to normalize the results:

[0112]

[0113]

[0114] in: is the final normalized brightness value, ranging from [0, 1]; is the brightness value of pixel i, is the weight of pixel i; T is the brightness threshold, preferably, 200 is used as the brightness threshold; S is the smoothing factor, preferably, 10 is used as the smoothing factor; k is the steepness parameter of the logistic function, preferably, 0.1 is used as the steepness parameter; μ is the center point of the logistic function, preferably, 128 is used as the center point = normalized local contrast of the active area; is the normalized local entropy of the effective area; p is the influencing factor of contrast and entropy, preferably, 0.5 is used as the influencing factor.

[0115] The value range of the image brightness reference value can be [0.65, 0.85]. In the embodiment of the present invention, 0.75 is given as the image brightness reference value, and the image whose brightness calculation result deviates too much from the reference value is brightened / darkened until it is close to the brightness reference value. Preferably, an iterative method is used to automatically find the available brightness adjustment factor, wherein a one-dimensional optimization method based on golden section search can be used to find the minimum value of the objective function by iteratively narrowing the search interval, thereby making the image brightness close to the brightness reference value.

[0116] Step 3: Get the coordinates of the F-TR lock image features to be synthesized. Use the container corner fittings target detection model to detect the positions of all corner fittings in the container image. The container corner fittings recognition frames on a single side-by-side image should show an obvious double-layer distribution pattern, with 4 (double-box side-by-side loading) or 2 (single-box loading) corner fittings recognition frames on the top and bottom. Take the lower layer recognition frame, such as Fig.11 shown.

[0117] Step 4: Synthesize the F-TR lock feature to the corner piece position. The synthesis process is divided into three steps, which are described in detail below:

[0118] Step 4.1: Preprocess the F-TR lock image. Randomly select the F-TR lock image extracted in the first stage, use the normalized brightness calculation formula proposed in Step 2 to calculate the brightness value of the container image to be synthesized and the brightness value of the F-TR lock image, and adjust the brightness value of the F-TR lock image until the difference between the two is .

[0119] Step 4.2: Overlay the F-TR lock image onto the corner piece identification frame. Align the lower left corner of the F-TR lock image with the lower left corner coordinates of the corner piece identification frame, perform layer overlay operation, and calculate the coordinate frame of the F-TR lock image on the new container image.

[0120] Step 4.3: Reconstruct the image using the gradient domain fusion method. In Step 4.1, directly superimposing layers will result in unnatural transitions around the F-TR locked image, and the model may learn incorrect features. Therefore, the gradient domain fusion method is used to eliminate the unnatural transitions and enhance the model recognition effect. The reconstruction effect is as follows: Fig.12 shown.

[0121] Step 5: Cut the container edge image. Use the container edge target detection model to predict the edge position of the container on the reconstructed image, and use the F-TR lock image coordinate frame in Step 4.2 for auxiliary calibration. If the edge prediction result cannot completely cover the X-axis coordinates of the left and right sides of the F-TR lock, expand the boundary coordinate value of the container edge position prediction frame until it can be completely covered, and save the image within the recognition frame.

[0122] Step 6: Calculate the F-TR lock coordinates after cutting the container edge image. For each F-TR lock coordinate frame, obtain the left boundary coordinate value of the container edge position prediction frame on the original image, recorded as , subtract the left boundary coordinate value from the horizontal coordinates of each vertex in the current F-TR lock coordinate frame , the relative horizontal coordinates of the F-TR lock coordinate frame in the cropped container edge image can be obtained.

[0123] Step 7: Perform model training. Use the cut container edge image and the F-TR lock coordinate frame as the coordinate frame of the object to be identified, and train the target detection model based on YOLOv8. During training, set the learning rate to 0.001, the batch size to 8, the mosaic data enhancement to 1.0, and the bounding box regression loss weight to 0.1.

[0124] 3. Data inference phase based on image feature adaptation, the main implementation steps include:

[0125] Step 1: Obtain high-definition vehicle images. Use a three-way linear array camera to collect images of vehicles that need to be identified.

[0126] Step 2: Use the vehicle type classification model to filter out the images of flatbed trucks loaded with containers and prepare for identification.

[0127] Step 3: Use the container edge target detection model to detect the filtered vehicle images, obtain the container edge area image, and capture its lower half as the final container edge image.

[0128] Step 4: Use the normalized brightness calculation formula proposed in Step 2 to calculate the brightness value of the container edge image. , adjust the image brightness until it differs from the reference value .

[0129] Step 5: Use the model trained in the second stage, Step 7, to perform detection, set the confidence level to 0.75, and obtain the F-TR lock feature detection results, such as Fig.13 shown.

[0130] Fig.14 FIG. 1 is a block diagram of a railway container corner fitting abnormal state detection system according to an exemplary embodiment of the present application. Fig.14 As shown, a railway container corner fitting abnormal state detection system includes an F-TR lock image acquisition module, a training data synthesis module, a model training module and a state detection module.

[0131] The F-TR lock image acquisition module is used to acquire a set of railway freight empty train images, and extract the F-TR lock image in each empty train image in the empty train image set based on an image segmentation algorithm;

[0132] A training data synthesis module is used to obtain a railway freight container passing image set, and to superimpose each F-TR lock image on a preset corner position of each container passing image in the container passing image set based on target detection and gradient threshold fusion to obtain a training data set;

[0133] A model training module is used to perform model training based on a training data set to obtain a target detection model with the F-TR lock as the detection target;

[0134] The state detection module is used to perform F-TR lock detection on the image of the container passing through the vehicle to be identified based on the target detection model, and determine the abnormal state of the container corner fittings according to the detection results.

[0135] An embodiment of the present invention further provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the method for detecting abnormal state of railway container corner fittings provided in any of the above embodiments.

[0136] An embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for detecting abnormal state of corner fittings of railway containers provided in any of the above embodiments is implemented.

[0137] The embodiment of the present invention further provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the abnormal state detection method of the railway container corner fitting provided in the above various optional embodiments.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0141] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0143] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting abnormal state of corner fittings of railway containers, characterized in that: include: Acquire a railway freight empty train image set, and extract an F-TR lock image from each empty train image in the empty train image set based on an image segmentation algorithm; Obtain a railway freight container passing image set, and superimpose each of the F-TR lock images on a preset corner piece position of each container passing image in the container passing image set based on target detection and gradient threshold fusion to obtain a training data set; Performing model training based on the training data set to obtain a target detection model with the F-TR lock as the detection target; Based on the target detection model, an F-TR lock detection is performed on the image of the container passing through the vehicle to be identified, and the abnormal state of the container corner fittings is determined according to the detection result; The method of acquiring a railway freight empty train image set and extracting an F-TR lock image from each empty train image in the empty train image set based on an image segmentation algorithm includes: S11: Obtain an image set of empty railway freight trains. Specifically, obtain loading status images of railway freight trains without containers through a linear array camera, and collect a large number of empty flat cars and empty container dedicated flat cars data set samples; S12: Cutting a local image of the F-TR lock position from each empty train image in the railway freight empty train image set. Specifically, taking the loading state images of the railway trains without containers on the left and right sides, for each side image I, taking the coordinates of the upper left corner of the image as point (0,0), taking three rectangular areas: Extract three rectangular areas to obtain a small area of ​​local image containing F-TR lock features; S13: extracting the F-TR lock polygonal image from the local image, specifically, using image editing software to process the local image, marking along the contour features of the F-TR lock, and obtaining polygonal contour marking data of each F-TR lock; S14: Using the local image to train an instance segmentation model, specifically, based on the YOLOv8-seg model, using the polygonal contour annotation data of the F-TR lock, based on the local image as the background image, the instance segmentation model is trained for F-TR lock feature extraction on the empty flat car image; S15: Use the instance segmentation model to cut all the empty vehicle passing images to obtain the F-TR lock image.

2. The method according to claim 1, characterized in that The method of superimposing each of the F-TR lock images to the preset corner position of each of the images of the vehicle with box passing in the vehicle with box passing image set based on target detection and gradient threshold fusion to obtain a training data set includes: Determine the preset corner fitting position of the container in each container passing image in the container passing image set based on the target detection algorithm; Align the lower left corner of the F-TR lock image to be superimposed with the lower left corner of the preset corner piece position in the container passing vehicle image to be superimposed to obtain the corresponding superimposed layer; Reconstructing the image of the superimposed layer based on a gradient domain fusion algorithm to obtain a corresponding synthetic image; extracting a container image from the composite image based on a target detection algorithm, and determining relative coordinates of the F-TR lock image on the container image; All the obtained container images constitute the training data set.

3. The method according to claim 2, characterized in that The determining of the preset corner fitting position of the container in each container-carrying vehicle image in the container-carrying vehicle image set based on the target detection algorithm comprises: Based on the container corner fittings target detection model, all corner fittings positions in the current container passing image are detected; The position of the lower layer identification box in the corner piece identification box presented in the double layer distribution mode in the detection result is recorded as the preset corner piece position.

4. The method according to claim 2, characterized in that: Before determining the preset corner fitting position of the container in each container-carrying vehicle image in the container-carrying vehicle image set based on the target detection algorithm, the method further includes: Determine a first brightness value of each image of a box-carrying vehicle by using a normalized brightness calculation method, and adjust the first brightness value until the difference between the first brightness value and a preset brightness reference value is less than a preset threshold value; A normalized brightness calculation method is used to determine a second brightness value of the F-TR lock image to be superimposed, and the second brightness value is adjusted until the difference between the second brightness value and the first brightness value of the container passing image to be superimposed is less than a preset threshold.

5. The method according to claim 4, characterized in that The normalized brightness calculation method is as follows: In the formula, is the final normalized brightness value, ranging from [0, 1], is the brightness value of pixel i, is the weight of pixel i, T is the brightness threshold, S is the smoothing factor, k is the steepness parameter of the logistic function, μ is the center point of the logistic function, is the normalized local contrast of the effective area, is the normalized local entropy of the effective area, and p is the influencing factor of contrast and entropy.

6. The method according to claim 2, characterized in that The extracting the container image in the synthetic image based on the target detection algorithm includes: Using a container edge target detection model to predict the edge position of the container on the synthetic image, to obtain an edge position prediction result; Using the F-TR locked image coordinates in the overlay layer to assist in calibrating the edge position prediction result; If the edge position prediction result cannot completely cover the X-axis coordinates of the left and right sides of the F-TR lock image, the boundary coordinate values ​​of the container edge position prediction box are expanded until they can be completely covered; The image within the container edge position prediction box is used as the container image.

7. The method according to any one of claims 1 to 6, characterized in that: The method of performing F-TR lock detection on the image of the container passing through the vehicle to be identified based on the target detection model and determining the abnormal state of the container corner fittings according to the detection result includes: Acquire an image of a vehicle with a container to be identified, and extract an image of an edge portion of a lower half of an image of a container in the image of a vehicle with a container to be identified; Determine a third brightness value of the edge portion image by using a normalized brightness calculation method, and adjust the third brightness value until the difference between the third brightness value and a preset brightness reference value is less than a preset threshold; The target detection model is used to perform F-TR lock detection on the edge portion image after brightness adjustment, and the abnormal state of the container corner fitting is determined according to the detection result.

8. A railway container corner fitting abnormal state detection system, characterized in that: The system for implementing the abnormal state detection method of railway container corner fittings according to any one of claims 1 to 7 comprises: The F-TR lock image acquisition module is used to acquire a set of railway freight empty train images, and extract an F-TR lock image from each empty train image in the empty train image set based on an image segmentation algorithm; A training data synthesis module is used to obtain a railway freight container passing image set, and to superimpose each of the F-TR lock images on a preset corner position of each container passing image in the container passing image set based on target detection and gradient threshold fusion to obtain a training data set; A model training module, used to perform model training based on the training data set to obtain a target detection model with the F-TR lock as the detection target; The state detection module is used to perform F-TR lock detection on the image of the container passing through the vehicle to be identified based on the target detection model, and determine the abnormal state of the container corner fittings according to the detection results.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a computer, the computer is caused to execute the method for detecting abnormal state of corner fittings of railway containers according to any one of claims 1 to 7.

10. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for detecting abnormal state of corner fittings of railway containers as described in any one of claims 1 to 7 when executing the program.

Images