Tamping lower pick area discrimination method, electronic device and computer readable storage medium

Abstract

The application discloses a tamping lower pick area discrimination method, electronic equipment and a computer readable storage medium. The method comprises the following steps: collecting an image of a lower pick area; dividing the image into subgraphs according to a set resolution; judging whether the current lower pick position is available for picking and whether there is another lower pick position according to the subgraphs; if a lower pick position is recorded, the fine adjustment distance in each fine adjustment direction is calculated according to the mapping relationship between the image and the real three-dimensional space coordinates; if no available lower pick position is recorded, it is determined that the lower pick position in the field of view is unavailable. The application can solve the technical problems that the existing tamping lower pick area discrimination method does not collect the real-time state of the tamping area, and does not consider the existence of obstacles, resulting in errors and omissions in the judgment of the lower pick area.

Description

Technical Field

[0001] This application relates to the field of railway engineering machinery technology, and is applied to large railway maintenance machinery, especially the method for identifying the tamping area of ​​mainline tamping machines and turnout tamping machines, electronic equipment and computer-readable storage media. Background Technology

[0002] A turnout tamping machine is a typical large-scale railway maintenance machine. It can be used for both railway turnout maintenance and mainline maintenance. It is mainly suitable for new line construction, post-major and minor maintenance cleaning of existing lines, and maintenance work on operating lines. Turnout tamping machines are used for automatic leveling, track lifting, and ballast tamping, improving the density of the ballast, increasing track stability, eliminating directional deviations, left and right horizontal deviations, and front and rear vertical deviations, ensuring the track meets the requirements of track design standards and maintenance rules, and guaranteeing the safe operation of trains. Tamping machines can be classified according to the number of sleepers tamped simultaneously (single-sleeper, double-sleeper, and four-sleeper tamping machines); according to the object being tamped (track and turnout tamping machines); according to the mode of operation (walking and continuous walking tamping machines); and according to their function (multi-functional and single-functional tamping machines). There are also tamping machines with special functions such as dust prevention and noise reduction.

[0003] Currently, turnout tamping machine operations still primarily rely on manual control. During operation, the primary operator must manually observe and determine the tamping position, continuously operating the tamping device's descent and the foot pedal for travel. Additionally, a secondary operator outside the machine must manually confirm the positions of the inner and outer tamping picks, prompting the primary operator to move the tamping frame outwards and rotate the picks. All of this requires operators to be highly skilled in equipment operation and maintain a high level of concentration, resulting in significant workload. Existing tamping operation control methods rely on real-time observation of the track conditions by personnel both inside and outside the machine to determine the area where the tamping device can be lowered. The relative relationship between the pick and objects near the drop point needs to be estimated manually, potentially leading to errors that could damage cables, railings, sleepers, or even rails. Furthermore, even after the tamping device reaches the drop point and stops, there may still be situations where the pick cannot be lowered, requiring multiple manual adjustments.

[0004] In existing technologies, some patents utilize image information to guide tamping operations. For example, Chinese invention applications CN116695501A and CN117557519A both use large-area track images, processed to obtain the positions of key objects such as sleepers, rails, and cables. However, the raw data is not the real-time state of the tamping area, and may contain errors and omissions. Furthermore, its purpose is only to help the operation control system plan work points and tamping strategies in advance, not specifically for tamping safety. Another example is Chinese invention application CN112376335A, which also uses image acquisition equipment to collect and detect sleeper positions in advance, and then guides the vehicle to lower the tamping pick based on the detected sleeper positions. This also suffers from the technical flaw of not being the real-time state of the tamping area, and potentially containing errors and omissions. For example, Chinese invention application CN118792921A places a video acquisition device in the middle of the tamping device to collect the status of the tamping point in real time. However, it only detects the center and edge of the sleeper to guide the insertion of the tamping device, but does not take into account the situation where there are obstacles. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a method for identifying the tamping pick area, an electronic device and a computer-readable storage medium, so as to solve the technical problems of existing tamping pick area identification methods not collecting the real-time status of the tamping area and not considering the presence of obstacles, which leads to errors and omissions in the identification of the pick area.

[0006] To achieve the aforementioned objectives, this application specifically provides a technical implementation scheme for a method for identifying the tamping pick area, comprising the following steps: S1) Acquire images of the lower pick area. ; S2) Image Divide into sub-images according to the set resolution. The set of the subgraphs is denoted as ; S3) Processing sets All subgraphs where the middle and lower pickaxe positions intersect The pixel-level segmentation of the area where the pick can be placed near the pick position is obtained. Then, the number of pixels that can be placed inside the pick position is counted. If the proportion of the area that can be placed inside is greater than the set threshold, it is determined that the current pick position can be placed. S4) If it is determined that the pickaxe cannot be placed, then process the set. All subgraphs not processed in step S3) Obtain each subgraph Determining whether a location is suitable for placing a pickaxe; S5) Obtaining the subgraph After determining whether a location is suitable for placing a pickaxe, the pickaxe placement location will be set in the image. Slide in the middle, and at each position, determine whether the pick can be placed according to step S3). If the pick can be placed, record the current position. S6) If a position where the pickaxe can be placed is recorded, the fine-tuning distance in each fine-tuning direction is calculated based on the mapping relationship between the image and the real three-dimensional space coordinates; if no position where the pickaxe can be placed is recorded, it is determined that the pickaxe cannot be placed within the field of view.

[0007] Furthermore, step S2) includes: S21) Calculate the image Number of subgraphs in the width direction Using images width Divide by the width of the box Round the result to the nearest integer. ; S22) Calculate the coordinates of the top-left corner of the subgraph in the width direction. Subgraphs are evenly divided Length, average spacing is The integer part is denoted as . , No. The coordinates of the top left corner of the subgraph are: ; S23) For images Repeat steps S21) and S22) in the height direction. S24) By combining the coordinates of all width and height directions, the two-dimensional coordinates of the top left corner of all sub-graphs can be obtained, and then the sub-graphs can be extracted based on these coordinates.

[0008] Furthermore, in step S3), the trained deep learning segmentation model is used to process the set. All subgraphs where the middle and lower pickaxe positions intersect This yields pixel-level segmentation of the area near the pickaxe's lowering position.

[0009] Furthermore, the training process of the deep learning semantic segmentation model in step S3) includes the following steps: S31) Acquire image data; S32) Based on the definition of the pickaxeable area, annotate all images using the annotation tool; S33) Simultaneously perform distortion correction and transformation into a top view on the original image and its mask image respectively. According to step S2), the original image, the distorted image, and the top view are segmented and each sub-image is saved. S34) In each image segmented sub-image, randomly select images of a set proportion to form a training set, and the remaining images to form an evaluation set; S35) Train a deep learning semantic segmentation model using the prepared training set, repeatedly adjust parameters, train, evaluate, and readjust parameters to select the optimal model; S36) Deployment Model.

[0010] Furthermore, in step S4), the trained deep learning classification model is used to process the set. E All subgraphs not processed in step S3) .

[0011] Furthermore, the training process of the deep learning classification model in step S4) includes the following steps: S41) Acquire image data; S42) Extract each sub-image from the acquired image according to the set resolution and save them as separate image files; S43) Check each sub-image and place the images into the corresponding category folders according to the criteria for pickaxe placement to achieve image annotation; S44) Randomly select a set proportion of images from each category to form the training set, and the remaining images to form the evaluation set; S45) Train the deep learning classification model using the prepared training set, repeatedly adjust the parameters, train, evaluate, and readjust the parameters to select the optimal model; S46) Deployment Model.

[0012] Furthermore, in step S6), the mapping process from the image to real three-dimensional spatial coordinates includes the following steps: S61) Detecting images midpoint The coordinates are ; S62) Based on the camera imaging model and intrinsic parameters, inversely calculate the points. In the normalized plane The coordinates in the middle, that is:

[0013] S63) Calculate the point in vector form, based on the geometric relationships of the point on the plane. The true three-dimensional spatial coordinates, that is:

[0014] S64) Find the point 3D coordinates:

[0015] in:

[0016] point The coordinates are ,point The coordinates are Rail base plane On point Projected onto the midpoint of the image The formula is:

[0017] Rail base plane The equation in the camera coordinate system is:

[0018] Camera internal parameters are ,Right now: .

[0019] Furthermore, in step S6), the calculation process for the fine-tuning distance of the tamping pick includes the following steps: S61) Use a deep learning classification model to process the sub-images that did not participate in the segmentation process, obtain the determination of whether each sub-image can be segmented, and uniformly mark all pixels in the sub-image as the determination. S62) is combined with the segmentation mask to form a large mask image of the original image size. After distortion correction and top view transformation, it becomes a mask image from a top view perspective. ; S63) Perform the same transformation on the calibrated lower pick position to obtain the result in the mask image. The area within ; S64) Traverse the region Determine the area The boundary coordinates, i.e., the left boundary Right boundary Upper boundary Lower boundary Establish coverage area mask subgraph Mask subgraph The width is Gao Wei ,area The pixel value within the specified area is set to 1, and the rest are set to 0. S65) in the mask image Slide within range Using the same benchmark as in step S3), determine whether the pick can be placed. If it can be placed, record the mask sub-image. In the mask image Position coordinates in; S66) Based on the mapping relationship between pixel coordinates and real 3D coordinates, calculate the 3D coordinates corresponding to the position recorded in step S65), thereby calculating the difference vector between the position and the original position. After projecting the difference vector onto the set fine-tuning direction, it is used as the fine-tuning distance in each fine-tuning direction. S67) Complete the entire mask image After the sliding search, if the fine-tuning in step S66 exists, the fine-tuning distance can be sent in subsequent processes; otherwise, a no-down-scraping signal is sent in subsequent processes.

[0020] This application also provides a specific technical implementation of an electronic device, which includes a memory and a processor. The memory stores program instructions, and when the processor runs the program instructions, it executes the steps of the method described in any one of claims 1-8.

[0021] This application also provides a specific technical implementation of a computer-readable storage medium, wherein the readable storage medium stores computer program instructions, which, when executed by a processor, perform the steps of the method described in any one of claims 1-8.

[0022] By implementing the above-described method for identifying the tamping area provided in this application, the technical solution of the electronic device and computer-readable storage medium has the following beneficial effects: (1) The tamping pick area discrimination method of this application, electronic equipment and computer readable storage medium, directly and in real time collect the state of the pick point and its vicinity through image acquisition device, and after processing by computer vision technology, identify various objects that affect the pick, calculate the relative positional relationship between the object and the tamping head landing point, and realize safe picking through control strategy judgment, realize the automatic judgment of safe picking, reduce unnecessary equipment damage, and also reduce the labor intensity of operators; (2) The tamping pick area discrimination method of this application, electronic device and computer readable storage medium, the detection method based on deep neural network can identify objects in the image. After the parameters between the track plane and the imaging device are calibrated in advance, the relative position relationship of the object in the actual scene can be calculated from the image, so as to calculate the distance required for the tamping pick to move to the tamping area, realize the automatic calculation of tamping pick fine adjustment, reduce repeated trial and error, and improve work efficiency. (3) The tamping pick area discrimination method of this application, electronic device and computer readable storage medium, adopts a method based on deep learning semantic segmentation model and image classification model to determine whether each sub-image of the image can be tamped, thereby restoring the tamping pick area segment of the corresponding original image. At the same time, based on the segmentation results, the coordinates of the image points on the real rail bottom plane can be calculated according to the camera imaging model and calibration results, which guides the fine adjustment of the tamping device. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the main program of a specific embodiment of the tamping pick area discrimination method of this application; Figure 2 This is a flowchart of the process for identifying and locating the pick-down area in a specific embodiment of the tamping pick-down area determination method of this application; Figure 3 This is a flowchart of the deep learning classification model training process in a specific embodiment of the tamping pick area discrimination method of this application; Figure 4 This is a flowchart of the deep learning segmentation model training process in a specific embodiment of the tamping pick area discrimination method of this application; Figure 5 This is a schematic diagram illustrating the principle of the image-to-real three-dimensional spatial coordinate mapping process in a specific embodiment of the tamping pick area discrimination method of this application; Figure 6 This is a flowchart of the tamping pick fine-tuning distance calculation process in a specific embodiment of the tamping pick area discrimination method of this application; Figure 7 This is a system structure block diagram of a specific embodiment of the tamping pick area discrimination device on which the method of this application is based; Figure 8 This is a system structure block diagram of a specific embodiment of the tamping pick area discrimination system on which the method of this application is based.

[0025] In the figure: 1-Image acquisition unit, 2-Tamping operation pick area discrimination device, 3-Operation control unit, 4-Tamping pick, 5-Camera, 21-Sub-image extraction unit, 22-Tamping judgment unit, 23-Tamping pick fine-tuning calculation unit, 24-Message sending unit, 221-Classification model processing unit, 222-Segmentation model processing unit. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] As attached Figure 1 To be continued Figure 8 The accompanying drawings illustrate specific embodiments of the tamping pick area discrimination method, electronic device, and computer-readable storage medium of this application. The application will be further described below with reference to the accompanying drawings and specific embodiments.

[0028] Example 1 In order to achieve image-based pickaxe safety determination and pickaxe position adjustment, this embodiment needs to solve the following two problems: 1) Identify and locate the areas in the image where the pick can be lowered, and avoid the presence of objects that may affect the lowering of the pick, including sleepers, rails, switch rails, guard rails, switches, gauge bars, turnouts, electrical cables, top rails, etc.

[0029] 2) Establish a spatial mapping between the image and the real scene. The mapping from image coordinates to world coordinates needs to be restored from two dimensions to three dimensions. This mapping is determined by the relative pose of the imaging device and the orbital plane.

[0030] At the same time, it is necessary to strictly control the computation time of the entire process from data collection to processing and output.

[0031] To address the first problem mentioned above, this embodiment employs a method combining image block processing with deep learning segmentation and classification models to accelerate processing speed while maintaining accuracy and precision. The specific steps of this embodiment are as follows: In this embodiment, an image acquisition unit 1 installed at the lower end of the tamping pick frame collects real-time scene data of the tamping area. The image data is processed by a tamping operation tamping area discrimination device 2 to identify the tamping area. This data is then compared with the tamping area in a pre-calibrated image to determine whether tamping is permissible. The judgment result is sent to the operation control unit 3. If tamping is not possible at the current location, the obtained tamping area segmentation and classification results can be used to determine whether there are other tamping locations between the sleepers. If so, the distance the vehicle should move is calculated based on the mapping relationship between the image coordinates and the real three-dimensional coordinates. If not, tamping is considered impossible.

[0032] As attached Figure 1 and appendix Figure 2 As shown, an embodiment of the method for identifying the tamping area according to this application specifically includes the following steps: S1) The imaging field of view of the image acquisition device includes the area where the pick is inserted, and images of the area where the pick is inserted are acquired. The resolution is ; S2) Image According to the set resolution Divide into subgraphs The set of subgraphs is denoted as The partitioning of subgraphs should ensure that the image... No pixels were not extracted; S3) Process the dataset using a trained deep learning semantic segmentation model. All subgraphs where the middle and lower pickaxe positions intersect The pixel-level segmentation of the area where the pick can be placed near the pick position is obtained. Then, the number of pixels that can be placed inside the pick position is counted. If the proportion of the area that can be placed inside is greater than the set threshold, it is determined that the current pick position can be placed. S4) If it is determined that the pickaxe cannot be placed, then use the trained deep learning classification model to process the set. All subgraphs not processed in step S3) Obtain each subgraph Determining whether a location is suitable for placing a pickaxe; S5) Obtain the subgraph through step S4) After determining whether a location is suitable for placing a pickaxe, the pickaxe placement location will be set in the image. Slide in the middle, and at each position, determine whether the pick can be placed according to step S3). If the pick can be placed, record the current position. S6) If a position where the pickaxe can be placed is recorded through step S5), the fine-tuning distance in each fine-tuning direction is calculated based on the mapping relationship between the image and the real three-dimensional space coordinates; if no position where the pickaxe can be placed is recorded through step S5), it is determined that the pickaxe cannot be placed within the field of view. S7) Send the information from step S3) indicating whether the pick can be lowered, or the information from step S6) indicating the fine-tuning distance, or the information from step S6) indicating whether the pick cannot be lowered, to the control terminal according to the communication protocol.

[0033] In step S2), the extraction of sub-images must ensure no pixels are missed, while minimizing the number of sub-images. Therefore, the following fine-tuning method after integer division is adopted, i.e., step S2) further includes: S21) Calculate the image Number of subgraphs in the width direction Using images width Divide by the width of the box Round the result to the nearest integer. ; S22) Calculate the coordinates of the top-left corner of the subgraph along the width direction. Along this axis, the coordinates of the top-left corner of the first subgraph are 0, and the coordinates of the top-left corner of the last subgraph are... ; Subgraphs are evenly divided Length, average spacing is The integer part is denoted as . Therefore, the first The coordinates of the top left corner of the subgraph are: ; S23) For images Repeat steps S21) and S22) in the height direction. S24) By combining the coordinates of all width and height directions, the two-dimensional coordinates of the top left corner of all sub-graphs can be obtained, and then the sub-graphs can be extracted based on these coordinates.

[0034] As attached Figure 3 As shown, in step S4), the trained deep learning classification model is used to process the set. E All subgraphs not processed in step S3) In this embodiment, the deep learning classification model can be PPHGNetV2, etc., and the categories are divided into two types: those that can be laid and those that cannot. The image corresponding to "those that can be laid" is completely filled with ballast or the ballast occupies more than 95% of the image; the rest is considered "those that cannot be laid". The training process of the deep learning classification model in step S4) includes the following steps: S41) Collect image data; ensure that the training data in the early stage is similar to the data collected when the final system is deployed in terms of viewpoint, scene, etc.; for example, install the image acquisition unit 1 on the bottom edge of the tamping frame and then walk on the actual line. S42) Process the data; extract each sub-image from the acquired image according to the set resolution (e.g., 128×128) and save them as separate image files; S43) Labeling; Check each sub-image and place the images into the corresponding category folders according to the aforementioned criteria for pickaxe placement to achieve image labeling; S44) Divide the training set and the evaluation set; randomly select a set proportion (e.g., 80%) of the images from each category to form the training set, and the remaining images to form the evaluation set; S45) Training the model; The deep learning classification model is trained using the prepared training set. The parameters can be adjusted repeatedly: training, evaluating, and readjusting the parameters to select the optimal model. S46) Deploy the model. Embed the pre- and post-processing and runtime code of the model into the software.

[0035] As attached Figure 4 As shown, in step S3), the trained deep learning segmentation model is used to process the set. All subgraphs where the middle and lower pickaxe positions intersect The pixel-level segmentation of the area near the pickaxe placement position is obtained. In this embodiment, the deep learning semantic segmentation model can use PPMobileSeg, etc., with only two categories: pickaxe-capable and pickaxe-uncapable. The pickaxe-capable region is the ballast area in the image; the remaining regions are pickaxe-uncapable. The training process of the deep learning semantic segmentation model in step S3) includes the following steps: S31) Acquire image data; requirements are the same as described in S41 above; S32) Annotation; Based on the definition of the pickaxe-capable region, annotate all images using the annotation tool; S33) Generate training data; simultaneously perform distortion correction and transformation into a top view on the original image and its mask image, and segment and save each sub-image according to the method in step S2); S34) Divide the image into a training set and an evaluation set; in each sub-image segmented from the image, randomly select a set proportion (e.g., 80%) of the images to form the training set, and the remaining images to form the evaluation set; S35) Training the model; The deep learning semantic segmentation model is trained using the prepared training set. The parameters can be adjusted repeatedly: training, evaluation, and parameter readjustment are performed to select the optimal model. S36) Deploy the model. Embed the pre- and post-processing and runtime code of the model into the software.

[0036] Regarding the second problem mentioned above, namely estimating the distance between the object within the track and the landing point of the pickaxe, which requires mapping a point in the image to a point in the real scene, this is called the extrinsic parameter calibration of camera 5. This embodiment derives the mathematical relationship between image coordinates and real 3D coordinates based on the pinhole imaging model of camera 5.

[0037] Assuming camera 5 produces distortion-free images, the process of projecting the real world onto the image can be simplified to the following: Figure 5 ,in It is the bottom plane of the rail. It is a virtual normalized plane of the pinhole imaging model. It is the imaging center of camera 5. (See attached image) Figure 5 As shown, in step S6), the mapping process from the image to the real three-dimensional spatial coordinates further includes the following steps: S61) Detecting images midpoint The coordinates are ; S62) Based on the camera imaging model and intrinsic parameters, inversely calculate the points. In the normalized plane The coordinates in the middle, that is:

[0038] S63) Calculate the point in vector form, based on the geometric relationships of the point on the plane. The true three-dimensional spatial coordinates, that is:

[0039] S64) Find the point 3D coordinates:

[0040] in:

[0041] Based on the pinhole imaging model of camera 5, the plane On point The coordinates are ,point The coordinates are Rail base plane On point Projected onto the midpoint of the image The formula is:

[0042] Rail base plane The equation in the camera's 5-coordinate system is:

[0043] Camera internal parameters are ,Right now: .

[0044] After determining whether tamping is feasible using the aforementioned method, to reduce the time consumed by repeated trial and error during manual adjustments, the tamping operation pick area discrimination device 2 can continue to provide the direction and distance required for pick fine-tuning. Assuming that the camera intrinsic parameters and distortion coefficients, and the equation of the rail base plane in the camera coordinate system have been calibrated, the steps for calculating the pick fine-tuning direction and distance are as follows: Figure 6 As shown, in step S6), the calculation process of the fine-tuning distance of the tamping pick further includes the following steps: S61) Use a deep learning classification model to process the sub-images that did not participate in the segmentation process, obtain the determination of whether each sub-image can be segmented, and uniformly mark all pixels in the sub-image as the determination. S62) is combined with the segmentation mask to form a large mask image of the original image size. After distortion correction and top view transformation, it becomes a mask image from a top view perspective. ; S63) Perform the same transformation on the calibrated lower pick position to obtain the result in the mask image. The area within ; S64) Traverse the region Determine the area The boundary coordinates, i.e., the left boundary Right boundary Upper boundary Lower boundary Establish coverage area mask subgraph Mask subgraph The width is Gao Wei ,area The pixel value within the specified area is set to 1, and the rest are set to 0. S65) in the mask image Slide within range Using the same benchmark as in step S3), determine whether the pick can be placed. If it can be placed, record the mask sub-image. In the mask image Position coordinates in; S66) Based on the mapping relationship between pixel coordinates and real 3D coordinates, calculate the 3D coordinates corresponding to the position recorded in step S65), thereby calculating the difference vector between the position and the original position. After projecting the difference vector onto the set fine-tuning direction, it is used as the fine-tuning distance in each fine-tuning direction. S67) Complete the entire mask image After the sliding search, if the fine-tuning described in step S66) exists, the fine-tuning distance can be sent in subsequent processes. Otherwise, a "cannot lower pick" signal is sent in subsequent processes.

[0045] This embodiment applies to mainline tamping machines and turnout tamping machines in large track maintenance machinery, involving automated tamping operation control, particularly image detection-based safety detection of the tamping pick. This embodiment also provides a tamping pick safety monitoring function to prevent damage to track equipment, and can provide necessary information for tamping strategies, improving operational efficiency.

[0046] Example 2 An embodiment of an electronic device specifically includes a memory and a processor. The memory stores program instructions, and when the processor executes the program instructions, it performs the steps of the method described in Embodiment 1.

[0047] Example 3 An embodiment of a computer-readable storage medium is provided, wherein the readable storage medium stores computer program instructions, which, when executed by a processor, perform the steps of the method described in embodiment 1.

[0048] Example 4 As attached Figure 7As shown, an embodiment of a pickaxe area discrimination device for tamping operations specifically includes: Sub-image extraction unit 21 extracts the acquired image of the lower pick area. Divide into sub-images according to the set resolution. ; Tamping judgment unit 22, based on sub-diagram Determine whether the current pickaxe position is a suitable location to place the pickaxe, and whether there are other suitable locations to place the pickaxe; If the pickaxe fine-tuning calculation unit 23 records a position where the pickaxe can be placed, it calculates the fine-tuning distance in each fine-tuning direction based on the mapping relationship between the image and the real three-dimensional space coordinates; if no position where the pickaxe can be placed is recorded, it determines that the pickaxe cannot be placed within the field of view. The message sending unit 24 sends a signal indicating whether the pick can be lowered, a signal indicating whether the pick cannot be lowered, or a signal indicating the fine-tuning distance of the pick.

[0049] Tamping Judgment Unit 22 Processes Sub-graph Set All subgraphs where the middle and lower pickaxe positions intersect The process involves obtaining pixel-level segmentation of the possible pick-down area near the pick-down position, then counting the number of possible pick-down pixels within the pick-down position. If the proportion of the possible pick-down area within the position exceeds a set threshold, the current pick-down position is considered possible. If it is determined to be impossible, the set is then processed. All subgraphs that were not processed Obtain each subgraph Determining whether a location is suitable for placing a pickaxe. This is done after obtaining the submap. After determining whether a location is suitable for placing a pickaxe, the pickaxe placement location will be set in the image. Slide the mouse in the middle and check at each position whether it is possible to place the pickaxe. If it is possible, record the current position.

[0050] The tamping judgment unit 22 includes a classification model processing unit 221, which uses a trained deep learning segmentation model to process the set. All subgraphs where the middle and lower pickaxe positions intersect The pixel-level segmentation of the area near the pickaxe-landing position is obtained. The classification model processing unit 221 collects image data, extracts each sub-image from the collected image according to the set resolution, and saves them as separate image files; it examines each sub-image and places it into the corresponding category folder according to the pickaxe-landing criteria to achieve image labeling; it randomly selects a set proportion of images from each category to form a training set, and the remaining images form an evaluation set; it trains the deep learning classification model using the prepared training set, repeatedly adjusting parameters-training-evaluation-readjusting parameters to select the optimal model; and completes the deployment of the deep learning classification model.

[0051] The tamping judgment unit 22 includes a segmentation model processing unit 222, which uses a trained deep learning classification model to process the set. E All subgraphs that were not processed The segmentation model processing unit 222 acquires image data and annotates all images using an annotation tool according to the definition of pickaxeable regions. It simultaneously performs distortion correction and transforms the original image and its mask image into a top-down view. The original image, the distorted image, and the top-down view are then segmented and each sub-image is saved. From the sub-images segmented from each image, a set proportion of images are randomly selected to form a training set, and the remaining images form an evaluation set. The prepared training set is used to train the deep learning semantic segmentation model, repeatedly adjusting parameters, training, evaluating, and readjusting parameters to select the optimal model. This completes the deployment of the deep learning segmentation model.

[0052] Sub-image extraction unit 21 calculates the image Number of subgraphs in the width direction Using images width Divide by the width of the box Round the result to the nearest integer. Calculate the coordinates of the top-left corner of the subgraph along the width direction. Subgraphs are evenly divided Length, average spacing is The integer part is denoted as . , No. The coordinates of the top left corner of the subgraph are: For images The number of subgraphs in the height direction and the coordinates of the top-left corner are calculated. By combining the coordinates of all width and height directions, the two-dimensional coordinates of the top-left corner of all subgraphs are obtained. Then, the subgraphs are extracted based on these coordinates.

[0053] The tamping pick fine-tuning calculation unit 23 detects the image. midpoint The coordinates are Based on the camera imaging model and intrinsic parameters, the points are inversely calculated. In the normalized plane The coordinates in the middle, that is:

[0054] Calculate the point using vector form and based on the geometric relationships of the point on the plane. The true three-dimensional spatial coordinates, that is:

[0055] Find the point 3D coordinates:

[0056] in:

[0057] point The coordinates are ,point The coordinates are Rail base plane On point Projected onto the midpoint of the image The formula is:

[0058] Rail base plane The equation in the camera coordinate system is:

[0059] Camera internal parameters are ,Right now: .

[0060] The pickaxe fine-tuning calculation unit 23 uses a deep learning classification model to process sub-images that did not participate in the segmentation process, obtains a determination of whether each sub-image can be picked, and uniformly marks all pixels in the sub-image with this determination; it combines with the segmentation mask to form a large mask image of the original image size, and after distortion correction and top view transformation, it becomes a mask image from a top view perspective. Perform the same transformation on the calibrated lower pick position to obtain the result in the mask image. The area within ; Traverse the region Determine the area The boundary coordinates, i.e., the left boundary Right boundary Upper boundary Lower boundary Establish coverage area mask subgraph Mask subgraph The width is Gao Wei ,area The pixel values ​​within the mask are set to 1, and the rest are 0; in the mask image Slide within range Determine if the current position allows for a pickaxe to be placed. If it does, record the mask sub-image. In the mask image The position coordinates in the image are determined; based on the mapping relationship between pixel coordinates and real 3D coordinates, the 3D coordinates corresponding to the recorded position are calculated, thereby calculating the difference vector between the recorded position and the original position. This difference vector is then projected onto the set fine-tuning directions and used as the fine-tuning distance in each fine-tuning direction; this completes the entire mask image. After the sliding search, if fine-tuning is possible, the fine-tuning distance is sent; otherwise, a "cannot be picked down" signal is sent.

[0061] Example 5 As attached Figure 8 As shown, an embodiment of a tamping operation pick area discrimination system specifically includes: Image acquisition unit 1, operation control unit and tamping operation pick area discrimination device 2 as described in Example 4; Image acquisition unit 1 acquires images of the lower pick area. ; The tamping operation pick area discrimination device 2 determines the area based on the image. The system processes the data and outputs a signal indicating whether the pick can be lowered, a signal indicating whether the pick cannot be lowered, or a signal indicating the fine-tuning distance of the pick to the work control unit 3. The operation control unit 3 controls the tamping pick based on the tamping pick can be lowered signal, the tamping pick cannot be lowered signal, or the tamping pick fine-tuning distance signal.

[0062] In the description of this specification, the references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0063] In the description of this application, it should be noted that when an element is referred to as being "fixed to" or "set on" another element, it can be directly set on the other element or indirectly set on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0064] It should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0065] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" or "several" means two or more, unless otherwise explicitly specified.

[0066] Those skilled in the art will further recognize that the units and steps of the various examples described in connection with the specific embodiments of this application can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0067] The methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, software modules executed by a processor, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, various programmable logic devices, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. The processor executing the software module can be a central processing unit (CPU), embedded processor, microcontroller (MCU), digital signal processor (DSP), single-chip microcomputer, system-on-a-chip (SoC), programmable logic device, or any other form of device with control and processing functions known in the art.

[0068] It should be noted that the structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which this application can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and purposes that this application can produce, should still fall within the scope of the technical content disclosed in this application.

[0069] By implementing the tamping pick area discrimination method described in the specific embodiments of this application, the technical solutions of electronic devices and computer-readable storage media can produce the following technical effects: (1) The tamping pick area discrimination method described in the specific embodiments of this application, electronic device and computer readable storage medium, directly and in real time collect the state of the pick point and its vicinity through image acquisition device, after processing by computer vision technology, identify various objects that affect the pick, calculate the relative positional relationship between the object and the tamping head landing point, and realize safe picking through control strategy judgment, realize the automatic judgment of safe picking, reduce unnecessary equipment damage, and also reduce the labor intensity of operators; (2) The tamping pick area discrimination method described in the specific embodiments of this application, electronic device and computer readable storage medium, the detection method based on deep neural network can identify objects in the image, and after calibrating the parameters between the track plane and the imaging device in advance, the relative position relationship of the object in the actual scene can be calculated from the image, so as to calculate the distance required for the tamping pick to move to the tamping pick area, realize the automatic calculation of tamping pick fine adjustment, reduce repeated trial and error, and improve work efficiency; (3) The tamping pick area discrimination method described in the specific embodiments of this application, electronic device and computer readable storage medium, uses a method based on deep learning semantic segmentation model and image classification model to determine whether each sub-image of the image can be tamped, thereby restoring the tamping pick area segment of the corresponding original image. At the same time, based on the segmentation results, the coordinates of the image points on the real rail bottom plane can be calculated according to the camera imaging model and calibration results, guiding the fine adjustment of the tamping device.

[0070] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0071] The above description is merely a preferred embodiment of this application and is not intended to limit this application in any way. Although this application has been disclosed above with reference to preferred embodiments, it is not intended to limit this application. Any person skilled in the art can make many possible variations and modifications to the technical solutions of this application using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the spirit and technical essence of this application. Therefore, any simple modifications, equivalent substitutions, equivalent changes, and modifications made to the above embodiments based on the technical essence of this application without departing from the content of the technical solutions of this application shall still fall within the protection scope of the technical solutions of this application.

Claims

1. A method for identifying the tamping pick area, characterized in that, Includes the following steps: S1) Acquire images of the lower pick area. ; S2) Image Divide into sub-images according to the set resolution. The set of the subgraphs is denoted as ; S3) Processing sets All subgraphs where the middle and lower pickaxe positions intersect The pixel-level segmentation of the area where the pick can be placed near the pick position is obtained. Then, the number of pixels that can be placed inside the pick position is counted. If the proportion of the area that can be placed inside is greater than the set threshold, it is determined that the current pick position can be placed. S4) If it is determined that the pickaxe cannot be placed, then process the set. All subgraphs not processed in step S3) Obtain each subgraph Determining whether a location is suitable for placing a pickaxe; S5) Obtaining the subgraph After determining whether a location is suitable for placing a pickaxe, the pickaxe placement location will be set in the image. Slide in the middle, and at each position, determine whether the pick can be placed according to step S3). If the pick can be placed, record the current position. S6) If a position where the pickaxe can be placed is recorded, the fine-tuning distance in each fine-tuning direction is calculated based on the mapping relationship between the image and the real three-dimensional space coordinates; if no position where the pickaxe can be placed is recorded, it is determined that the pickaxe cannot be placed within the field of view.

2. The method for identifying the tamping pick area according to claim 1, characterized in that, Step S2) further includes: S21) Calculate the image Number of subgraphs in the width direction Using images width Divide by the width of the box Round the result to the nearest integer. ; S22) Calculate the coordinates of the top-left corner of the subgraph in the width direction. Subgraphs are evenly divided Length, average spacing is The integer part is denoted as . , No. The coordinates of the top left corner of the subgraph are: ; S23) For images Repeat steps S21) and S22) in the height direction. S24) By combining the coordinates of all width and height directions, the two-dimensional coordinates of the top left corner of all sub-graphs can be obtained, and then the sub-graphs can be extracted based on these coordinates.

3. The method for identifying the tamping pick area according to claim 1 or 2, characterized in that: In step S3), the trained deep learning segmentation model is used to process the set. All subgraphs where the middle and lower pickaxe positions intersect This yields pixel-level segmentation of the area near the pickaxe's lowering position.

4. The method for identifying the tamping pick area according to claim 3, characterized in that, The training process of the deep learning semantic segmentation model in step S3) includes the following steps: S31) Acquire image data; S32) Based on the definition of the pickaxeable area, annotate all images using the annotation tool; S33) Simultaneously perform distortion correction and transformation into a top view on the original image and its mask image respectively. According to step S2), the original image, the distorted image, and the top view are segmented and each sub-image is saved. S34) In each image segmented sub-image, randomly select images of a set proportion to form a training set, and the remaining images to form an evaluation set; S35) Train a deep learning semantic segmentation model using the prepared training set, repeatedly adjust parameters, train, evaluate, and readjust parameters to select the optimal model; S36) Deployment Model.

5. The method for identifying the tamping pick area according to claim 4, characterized in that: In step S4), the trained deep learning classification model is used to process the set. E All subgraphs not processed in step S3) .

6. The method for identifying the tamping pick area according to claim 5, characterized in that, The training process of the deep learning classification model in step S4) includes the following steps: S41) Acquire image data; S42) Extract each sub-image from the acquired image according to the set resolution and save them as separate image files; S43) Check each sub-image and place the images into the corresponding category folders according to the criteria for pickaxe placement to achieve image annotation; S44) Randomly select a set proportion of images from each category to form the training set, and the remaining images to form the evaluation set; S45) Train the deep learning classification model using the prepared training set, repeatedly adjust the parameters, train, evaluate, and readjust the parameters to select the optimal model; S46) Deployment Model.

7. The method for identifying the tamping area according to claim 1, 2, 4, 5 or 6, characterized in that, In step S6), the mapping process from the image to real three-dimensional spatial coordinates includes the following steps: S61) Detecting images midpoint The coordinates are ; S62) Based on the camera imaging model and intrinsic parameters, inversely calculate the points. In the normalized plane The coordinates in the middle, that is: ； S63) Calculate the point in vector form, based on the geometric relationships of the point on the plane. The true three-dimensional spatial coordinates, that is: ； S64) Find the point 3D coordinates: ； in: ； point The coordinates are ,point The coordinates are Rail base plane On point Projected onto the midpoint of the image The formula is: ； Rail base plane The equation in the camera coordinate system is: ； Camera internal parameters are ,Right now: 。 8. The method for identifying the tamping pick area according to claim 7, characterized in that, In step S6), the calculation process for the fine-tuning distance of the tamping pick includes the following steps: S61) Use a deep learning classification model to process the sub-images that did not participate in the segmentation process, obtain the determination of whether each sub-image can be segmented, and uniformly mark all pixels in the sub-image as the determination. S62) is combined with the segmentation mask to form a large mask image of the original image size. After distortion correction and top view transformation, it becomes a mask image from a top view perspective. ; S63) Perform the same transformation on the calibrated lower pick position to obtain the result in the mask image. The area within ; S64) Traverse the region Determine the area The boundary coordinates, i.e., the left boundary Right boundary Upper boundary Lower boundary Establish coverage area mask subgraph Mask subgraph The width is Gao Wei ,area The pixel value within the specified area is set to 1, and the rest are set to 0. S65) in the mask image Slide within range Using the same benchmark as in step S3), determine whether the pick can be placed. If it can be placed, record the mask sub-image. In the mask image Position coordinates in; S66) Based on the mapping relationship between pixel coordinates and real 3D coordinates, calculate the 3D coordinates corresponding to the position recorded in step S65), thereby calculating the difference vector between the position and the original position. After projecting the difference vector onto the set fine-tuning direction, it is used as the fine-tuning distance in each fine-tuning direction. S67) Complete the entire mask image After the sliding search, if the fine-tuning in step S66 exists, the fine-tuning distance can be sent in subsequent processes; otherwise, a no-down-scraping signal is sent in subsequent processes.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory storing program instructions, and the processor executing the steps of the method according to any one of claims 1-8 when running the program instructions.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores computer program instructions, which, when executed by a processor, perform the steps of the method according to any one of claims 1-8.

Images