Trolley wheel oil nozzle positioning method and system based on lightweight instance segmentation

Abstract

The application discloses a trolley wheel oil nozzle positioning method and system based on lightweight instance segmentation, and belongs to the field of trolley wheel oil nozzle positioning. The method specifically comprises the following steps: acquiring a trolley wheel oil nozzle positioning element set; based on the trolley wheel oil nozzle positioning element set, performing wheel image matching and instance segmentation to obtain a current semantic probability graph sequence, a historical semantic probability graph sequence and an instance mask sequence; based on a trolley wheel running image, the current semantic probability graph sequence, the historical semantic probability graph sequence and the instance mask sequence, constructing an axle heterogeneous motion semantic graph to obtain an oil nozzle stable structure semantic mask sequence and a common motion attached residual sequence; and based on the common motion attached residual sequence, the oil nozzle stable structure semantic mask sequence, the current semantic probability graph sequence and the instance mask sequence, determining an oil nozzle structure positioning center to obtain a trolley wheel oil nozzle positioning result. The application improves the accuracy of trolley wheel oil nozzle positioning and enhances the stability of trolley wheel oil nozzle positioning.

Description

Technical Field

[0001] This invention belongs to the field of trolley wheel grease nipple positioning technology, specifically relating to a trolley wheel grease nipple positioning method and system based on lightweight instance segmentation. Background Technology

[0002] During the positioning of the oil nozzles on the sintering trolley wheels, it is necessary to identify the oil nozzles from continuously acquired images of the trolley wheels and determine their positions. When the sintering trolley is running, the wheel body rotates continuously, and the shaft end and oil nozzles mainly move with the trolley. Moreover, grease and dust easily adhere to the area around the oil nozzles, resulting in differential motion and boundary interference in the continuous images, making it difficult to extract the oil nozzle positions stably.

[0003] In existing technologies, the trolley wheel nozzle localization method based on lightweight instance segmentation typically segments nozzle instances according to the appearance features and continuous motion characteristics of the entire trolley wheel region, and uses the centroid or circumscribed center of the instance mask as the localization result. However, the motion patterns of the wheel and the nozzle at the axle end are different, and the texture changes caused by the wheel rotation can interfere with the positional association of the nozzle in adjacent images. At the same time, grease and dust adhering to the nozzle will move synchronously with the nozzle and are easily continuously scraped into the nozzle instance mask, causing positional drift and boundary expansion of the nozzle mask, resulting in poor accuracy and stability of trolley wheel nozzle localization. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for locating trolley wheel nozzles based on lightweight instance segmentation. By constructing a heterogeneous motion semantic graph of the wheel axle and performing joint semantic inference of nozzle candidate pixels, the accuracy and stability of trolley wheel nozzle positioning are improved.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] On one hand, the present invention provides a method for positioning trolley wheel nozzles based on lightweight instance segmentation, including:

[0007] Obtain the positioning element set of the trolley wheel oil nozzle, which includes historical image data, lightweight instance segmentation network, trolley operation identification information and trolley wheel operation images;

[0008] Based on the positioning element set of the trolley wheel oil nozzle, image matching and instance segmentation of the same wheel are performed to obtain the current semantic probability map sequence, the historical semantic probability map sequence and the instance mask sequence.

[0009] Based on the trolley wheel running image, the current semantic probability map sequence, the historical semantic probability map sequence, and the instance mask sequence, a wheel-axle heterogeneous motion semantic map is constructed to obtain the nozzle stable structure semantic mask sequence and the common motion attachment residual sequence. The nozzle stable structure semantic mask sequence and the common motion attachment residual sequence are obtained by performing motion constraints and joint inference of the nozzle candidate pixels based on the wheel-axle heterogeneous motion semantic map.

[0010] Based on the common motion attachment residual sequence, the semantic mask sequence of the nozzle stable structure, the current semantic probability map sequence, and the instance mask sequence, the nozzle structure positioning center is determined, and the nozzle positioning result of the trolley wheel is obtained.

[0011] Specifically, based on the positioning feature set of the trolley wheel nozzles, image matching and instance segmentation of the same wheel are performed to obtain the current semantic probability map sequence, the historical semantic probability map sequence, and the instance mask sequence, including:

[0012] Based on the trolley operation identification information and the trolley wheel operation images, a trolley identification is generated, and the trolley wheel operation images are arranged to obtain the current image sequence of the trolley wheels.

[0013] Based on the historical image data and trolley identification, historical images of the same trolley wheels are extracted and image registration is performed to obtain a historical passing image sequence;

[0014] The current image sequence and the historical passing image sequence of the trolley wheels are input into the lightweight instance segmentation network for multi-scale feature extraction and fusion to obtain the current semantic probability map sequence, the historical semantic probability map sequence, the instance center response and the center offset.

[0015] The foreground pixels in the current semantic probability map sequence and the historical semantic probability map sequence are determined, and instances are grouped according to the instance center response and center offset to obtain the instance mask sequence.

[0016] Specifically, based on the trolley wheel running image, the current semantic probability map sequence, the historical semantic probability map sequence, and the instance mask sequence, a heterogeneous motion semantic map of the wheel and axle is constructed, resulting in a grout nozzle stable structure semantic mask sequence and a common motion attachment residual sequence, including:

[0017] Based on the trolley wheel running image, the current semantic probability map sequence and the instance mask sequence, the common translation of the axle end is extracted and the wheel body rotation phase is disambiguated to obtain the axle end translation flow field and the wheel body phase disambiguated rotation flow field.

[0018] Based on the trolley operation identification information, the current semantic probability map sequence, the historical semantic probability map sequence, the instance mask sequence, and the flow field, a wheel-axle heterogeneous motion semantic map is constructed.

[0019] Based on the heterogeneous motion semantic map of the wheel and axle and the flow field, candidate pixels of the nozzle are determined, and root continuation semantic clique and rotational traction semantic clique are constructed to obtain a high-order conditional random field.

[0020] Based on high-order conditional random fields, joint semantic inference and structural aggregation are performed on candidate pixels of the nozzle to obtain a stable structural semantic mask sequence and a common motion attachment residual sequence of the nozzle.

[0021] Specifically, based on the trolley wheel running image, the current semantic probability map sequence, and the instance mask sequence, the common translation of the axle end is extracted and the wheel rotation phase is disambiguated to obtain the axle end translation flow field and the wheel phase-disambiguated rotational flow field, including:

[0022] Bidirectional dense optical flow estimation is performed on the images of adjacent trolley wheels to obtain forward dense optical flow, reverse dense optical flow and bidirectional pixel correspondence confidence;

[0023] Extract the axle end instance region and wheel body instance region, and combine the axle end semantic probability and bidirectional pixel correspondence confidence to perform weighted aggregation and residual correction on the forward dense optical flow to obtain the axle end common translation amount and axle end translation flow field.

[0024] Extract the semantic boundary response and image gradient response of the wheel body within the wheel body instance region, and perform circumferential expansion and radial aggregation with the center of the axle end instance as the reference to obtain the circumferential semantic expansion sequence;

[0025] The circumferential semantic unfolding sequence of the next frame is corrected based on the common translation of the axle end, and cyclic correlation matching is performed with the circumferential semantic unfolding sequence of the previous frame to obtain the wheel rotation phase candidate and the degree of cyclic correlation.

[0026] By combining the forward dense optical flow, the reverse dense optical flow, and the common translation at the shaft end, the candidate rotating phase of the wheel body is verified by rotation and screened for continuity, resulting in the wheel body phase disambiguation rotating flow field.

[0027] Specifically, by combining the forward dense optical flow, the reverse dense optical flow, and the common translation at the shaft end, the candidate rotational phases of the wheel body are subjected to rotational verification and continuity screening to obtain the wheel body phase-disambiguated rotating flow field, including:

[0028] Based on the rotation phase candidates of each wheel body, the pixels in the wheel body instance area are rotated around the center of the shaft end instance and the common translation of the shaft end is superimposed to obtain the wheel body candidate motion flow field;

[0029] By comparing the candidate motion flow field of the wheel body with the forward dense optical flow, the candidate motion residuals are obtained, and the reverse dense optical flow is used for reverse mapping to obtain the difference between forward and reverse rotational closure.

[0030] Based on the candidate motion residuals, the difference in forward and reverse rotation closure, and the degree of cyclic correlation, the first group of target rotation phases is selected.

[0031] Based on the image order and the continuity difference between the current candidate and the previous group of target rotation phases, the remaining target rotation phases are filtered.

[0032] Based on the rotational phase of each group of targets and the common translation of the shaft end, the rotational translation mapping of the wheel body instance region is performed to obtain the wheel body phase-disambiguated rotating flow field.

[0033] Specifically, based on the trolley operation identification information, the current semantic probability map sequence, the historical semantic probability map sequence, the instance mask sequence, and the flow field, a heterogeneous motion semantic map of the wheel and axle is constructed, including:

[0034] Extract historical and current shaft end instances, wheel body instances, and oil nozzle candidate instances from the current semantic probability graph sequence, historical semantic probability graph sequence, and instance mask sequence, establish corresponding nodes, and obtain a heterogeneous semantic node set;

[0035] By mapping the historical shaft end instance using the shaft end translation flow field, and based on the degree of overlap and positional deviation between the mapped region and the current shaft end instance, a shaft end translation receiving edge is established;

[0036] By using the phase-disambiguated rotating flow field mapping of wheel body instances to map historical wheel body instances, and based on the degree of overlap and semantic distribution differences between the mapped region and the current wheel body instance, a wheel body rotation bearing edge is established.

[0037] Based on the trolley operation identification information, extract the historical connection root region and the current connection root region of the same trolley wheel, and establish the connection root reproduction edge through axle end benchmark alignment and reproduction comparison;

[0038] By organizing the heterogeneous semantic node set and various edges into a graph structure, a heterogeneous motion semantic graph of wheels and axles is obtained.

[0039] Specifically, based on the trolley operation identification information, the historical and current connection root regions of the same trolley wheels are extracted. After axle end datum alignment and reproduction comparison, connection root reproduction edges are established, including:

[0040] Based on the connection position between the nozzle candidate instance mask and the shaft end instance mask, extract the current connection root region;

[0041] Based on the trolley operation identification information, extract the historical oil nozzle candidate instance mask and historical axle end instance mask of the same trolley wheel from the historical semantic probability map sequence and instance mask sequence to determine the historical connection root region;

[0042] Determine the current axis-end geometric reference and the historical axis-end geometric reference based on the axis-end center, region size, and main axis direction of the current axis-end instance mask and the historical axis-end instance mask, respectively;

[0043] Construct the current axis-end local coordinate system based on the geometric datum, and map the historical connection root region to the local coordinate system to obtain the aligned historical connection root region;

[0044] Compare the current connection root region with the aligned historical connection root region to determine the degree of connection root repetition and establish connection root repetition edges.

[0045] Specifically, based on the heterogeneous motion semantic map of the wheel and axle and the flow field, candidate pixels for the nozzle are determined, a root continuation semantic clique and a rotational traction semantic clique are constructed, and a higher-order conditional random field is obtained, including:

[0046] Candidate pixels for the nozzle are determined from the current semantic probability map sequence and instance mask sequence. Semantic probabilities of the nozzle, shaft end and wheel body are extracted to obtain the set of pixels to be inferred.

[0047] Motion mapping of the pixel to be inferred is performed using the translational flow field at the shaft end and the phase-disambiguated rotating flow field of the wheel body to construct the single-pixel potential and the pixel-pair potential.

[0048] The association between the pixel to be inferred and the axle end instance is determined based on the heterogeneous motion semantic graph of the wheel and axle, and the axle end structure continuation pixels are collected to obtain the root continuation semantic group.

[0049] The association between the pixel to be inferred and the wheel instance is determined based on the heterogeneous motion semantic map of the wheel axle and the phase disambiguation rotating flow field of the wheel body. The rotating mapping continuous pixels are collected to obtain the rotating traction semantic group.

[0050] Based on two types of semantic groups, a connection root continuation potential and a wheel rotation repulsion potential are constructed, and combined with a single pixel potential and a pixel pair potential to obtain a higher-order conditional random field.

[0051] Specifically, based on the co-motion attachment residual sequence, the nozzle stable structure semantic mask sequence, the current semantic probability map sequence, and the instance mask sequence, the nozzle structure positioning center is determined, and the trolley wheel nozzle positioning result is obtained, including:

[0052] The positions of the co-motion attachment residual sequence and the nozzle stable structure semantic mask sequence are matched, and the corresponding co-motion attachment regions are removed to obtain the nozzle structure mask.

[0053] Extract the shaft end instance region from the instance mask sequence and determine the adjacent boundary between the nozzle structure mask and the shaft end instance region to obtain the nozzle shaft end connection boundary;

[0054] The contour of the nozzle structure mask is extracted and filtered by combining the current semantic probability map sequence to obtain the axial contour of the nozzle.

[0055] Based on the connection boundary of the nozzle shaft end and the axial contour of the nozzle, the center of the nozzle structure is calculated to obtain the positioning result of the nozzle on the trolley wheel.

[0056] On the other hand, the present invention provides a trolley wheel nozzle positioning system based on lightweight instance segmentation, comprising:

[0057] The element acquisition module is used to acquire the positioning element set of the trolley wheel oil nozzles;

[0058] The same-wheel matching module performs same-wheel image matching and instance segmentation based on the positioning element set of the trolley wheel oil nozzle, and obtains the current semantic probability map sequence, the historical semantic probability map sequence and the instance mask sequence;

[0059] The semantic graph construction module constructs a semantic graph of heterogeneous motion of wheel and axle based on the trolley wheel running image, the current semantic probability graph sequence, the historical semantic probability graph sequence and the instance mask sequence, and obtains the semantic mask sequence of the oil nozzle stable structure and the common motion attachment residual sequence.

[0060] The nozzle positioning module determines the nozzle structure positioning center based on the common motion attachment residual sequence, the nozzle stable structure semantic mask sequence, the current semantic probability map sequence, and the instance mask sequence, thus obtaining the nozzle positioning result of the trolley wheel.

[0061] Compared with the prior art, the beneficial effects of the present invention include: by constructing a semantic graph of heterogeneous motion of the wheel and axle, the translational bearing relationship of the axle end, the rotational bearing relationship of the wheel body, and the reproduction relationship of the oil nozzle axle end connection structure are associated and organized, thereby enhancing the stability of oil nozzle structure recognition; based on a high-order conditional random field, joint semantic inference is performed on oil nozzle candidate pixels, and the root continuation semantic clique and the rotational traction semantic clique are used to constrain the axle end connection structure maintenance relationship and the wheel body rotational attachment relationship, respectively, so that the real oil nozzle area and the attachment interference area that moves with the wheel body can be effectively distinguished, thereby improving the accuracy of trolley wheel oil nozzle positioning. Attached Figure Description

[0062] Figure 1 Flowchart of the trolley wheel oil nozzle positioning method based on lightweight instance segmentation provided by the present invention;

[0063] Figure 2 This is a schematic diagram of the construction of the root continuation semantic group provided by the present invention;

[0064] Figure 3 The diagram shows the structure of the trolley wheel oil nozzle positioning system based on lightweight instance segmentation provided by the present invention. Detailed Implementation

[0065] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0066] Example 1

[0067] See Figures 1-2 This embodiment provides a method for positioning trolley wheel oil nozzles based on lightweight instance segmentation, including the following specific steps:

[0068] Step S1: Obtain the positioning element set of the trolley wheel oil nozzle, which includes historical image data, lightweight instance segmentation network, trolley operation identification information and trolley wheel operation images.

[0069] In this embodiment, an image acquisition device positioned beside the trolley wheel's running path continuously acquires images of the trolley wheel as it passes through the image acquisition area, resulting in trolley wheel running images. The image acquisition device is an industrial camera, and its field of view covers the wheel body, axle end, and oil nozzle area. The trolley wheel running images include multiple consecutive frames of the same trolley wheel formed during a single passage through the image acquisition area.

[0070] The image acquisition frequency is set based on the trolley's maximum operating speed and the minimum visible width of the nozzle in the image. Specifically, the setting method is as follows: obtain the trolley's maximum operating speed within its normal operating range; determine half of the nozzle's minimum visible width as the maximum allowable nozzle displacement between adjacent images; divide the image displacement speed corresponding to the maximum operating speed by the maximum allowable nozzle displacement to obtain the minimum image acquisition frequency, ensuring that the nozzle displacement between adjacent images does not exceed half of the nozzle's minimum visible width. The number of images acquired in a single session is set based on the duration the trolley wheels are within the effective acquisition field of view and the image acquisition frequency. Specifically, multiply the duration by the image acquisition frequency and round the result up to obtain the number of images acquired in a single session.

[0071] Based on the operation record output by the trolley control equipment, the trolley number, wheel side identification, operation cycle, and image acquisition time are read to obtain trolley operation identification information. The trolley number is used to distinguish different trolleys, the wheel side identification is used to distinguish different wheels on the same trolley, the operation cycle is used to distinguish different passing processes of the same trolley wheel, and the image acquisition time is used to determine the acquisition order of the trolley wheel operation images.

[0072] The trolley number and wheel side are associated to obtain a unique identifier for each trolley wheel. The trolley wheel running images, the unique identifiers of the trolley wheels, the running cycle, and the image acquisition time are associated and stored in a historical image repository. Based on the unique identifiers of the trolley wheels, consecutive images of the same trolley wheel acquired before the current running cycle are extracted from the historical image repository and arranged according to the running cycle and image acquisition time to obtain historical image data.

[0073] The number of historical operating cycles is set based on the cross-cycle reproducibility stability of the wheel connection structure of the same vehicle and the timeliness of historical images. Specifically, in the offline verification data, the historical operating cycles of the same vehicle's wheels are arranged from most recent to oldest acquisition time. For any historical operating cycle, the corresponding historical connection root region is extracted, and based on the current axle end geometric reference and the historical axle end geometric reference, the historical connection root region is mapped to the current image coordinate range to obtain the aligned historical connection root region. The degree of regional overlap, center position consistency, connection direction consistency, and semantic distribution consistency between the current connection root region and the aligned historical connection root region are calculated. These four consistency factors are multiplied by their corresponding weights and summed to obtain the single-reproducibility degree of that historical operating cycle. The degree of regional overlap is calculated by dividing the number of pixels in the intersection of two connection root regions by the number of pixels in the union. The degree of consistency in center positions is obtained by normalizing the center distance between the two connection root regions to the axial end region size and then taking the negative exponent. The degree of consistency in connection direction is obtained by calculating the angle between the direction vectors of the two connection roots and subtracting the ratio of that angle to 180 degrees from 1. The degree of consistency in semantic distribution is obtained by subtracting the normalized difference in the semantic probability distribution of the nozzles within the two connection root regions from 1. The weights for each item are set according to the degree of distinction between real and non-connection roots. Specifically, the weight combination is adjusted under the condition that the sum of all weights is 1, selecting the weight combination with the largest difference between the average recurrence rate of real connection roots and the average recurrence rate of non-connection roots. For the first K historical operating cycles selected from the nearest to the furthest, the single recurrence rate of each historical operating cycle is added together and then divided by K to obtain the average recurrence rate of the connection root regions when the number of historical operating cycles is K. Starting from the second historical operating cycle, the average reproducibility corresponding to the current number of historical operating cycles is subtracted from the average reproducibility corresponding to the previous number of historical operating cycles, and the absolute value is taken to obtain the change in reproducibility. Historical operating cycles are gradually increased. When the change in reproducibility in the current cycle is not less than the change in reproducibility in the previous cycle, it indicates that continuing to introduce earlier historical images has not further improved the stability of cross-cycle reproducibility results. At this point, the addition of historical operating cycles is stopped, and the number of historical operating cycles included before the cessation of addition is determined as the adopted number.

[0074] A lightweight instance segmentation network was constructed based on labeled training images of trolley wheels. The training images included those with clean nozzle surfaces, grease buildup, dust buildup, partial nozzle occlusion, partial wheel reflection, and trolley wheel position offset. Each training image was labeled with the background region, wheel region, axle end region, and nozzle region, and independent instance labels were assigned to different wheel, axle, and nozzle instances.

[0075] The lightweight instance segmentation network comprises an encoding module, a feature fusion module, and an output module connected in sequence. The encoding module includes four feature extraction layers. Each feature extraction layer consists of a depthwise convolutional layer, a pointwise convolutional layer, a batch normalization layer, and an activation layer. The depthwise convolutional layer uses a 3×3 pixel kernel to extract spatial features from each input channel, while the pointwise convolutional layer uses a 1×1 pixel kernel to fuse different channels. Adjacent feature extraction layers are connected by a depthwise convolution with a stride of 2, halving the feature map resolution at each level.

[0076] The feature fusion module starts from the 4th level encoded features, performs bilinear upsampling of the current feature map at a ratio of 2, and then adjusts the number of channels to match the previous level encoded features through a 1 pixel × 1 pixel convolution. The upsampled features and the previous level encoded features are then concatenated by channel and fused through a 3 pixel × 3 pixel convolution. The fusion of features from the 4th level to the 1st level is completed in the same manner to obtain the fused feature map.

[0077] The output module includes a semantic prediction branch, an instance center prediction branch, and a center offset prediction branch. Each output branch is connected to the fused feature map. The semantic prediction branch consists of a 3-pixel × 3-pixel convolutional layer and a 1-pixel × 1-pixel output layer, outputting the category responses for four channels: background, wheel body, axle end, and nozzle. The probabilities of these four channels are then normalized to obtain the semantic probability of each pixel. The instance center prediction branch outputs the instance center responses for three channels: wheel body, axle end, and nozzle. The center offset prediction branch outputs two channels, representing the horizontal and vertical offsets of each foreground pixel towards the corresponding instance center, respectively.

[0078] When training the lightweight instance segmentation network, the training image is input into the encoding module and the feature fusion module. Prediction results are obtained through semantic prediction, instance center prediction, and center offset prediction branches, respectively. The semantic prediction results are compared pixel-by-pixel with the pixel category labels, and the average of the negative logarithms of the predicted category probabilities is calculated to obtain the semantic classification error. The number of intersection pixels between the predicted foreground region and the labeled foreground region is calculated. Twice the number of intersection pixels is divided by the sum of the predicted and labeled foreground pixel counts, and the result is subtracted from 1 to obtain the region overlap error.

[0079] The instance center error is obtained by subtracting the predicted instance center response from the labeled instance center response pixel by pixel, squaring the differences, and averaging them. The center offset error is obtained by subtracting the predicted center offset from the labeled center offset component by component, taking the absolute values ​​of the horizontal and vertical differences, and averaging them. The network training error is obtained by weighted summing of the semantic classification error, region overlap error, instance center error, and center offset error.

[0080] The weights of each training error are set based on the numerical scale of each training error in the initial training phase. Specifically, the weights are calculated as follows: the median of semantic classification error, region overlap error, instance centering error, and center offset error in the initial training batch is calculated, and the reciprocal of each median is normalized to obtain the initial weights for each training error. Using these initial weights as a baseline, one weight is increased or decreased, and all adjusted weights are renormalized to ensure the sum of all weights is 1. For each weight combination, validation training is performed using the same training images, training batches, and training periods. The nozzle segmentation overlap degree and instance centering error of the validation images are calculated. When the nozzle segmentation overlap degree no longer increases after continuous changes in the same adjustment direction, or when the instance centering error begins to increase, the weight adjustment in that direction is stopped. The weight combinations formed before the adjustment was stopped are summarized, and the weight combination with the highest nozzle segmentation overlap degree is selected. When multiple weight combinations have the same nozzle segmentation overlap degree, the weight combination with the smallest instance centering error is selected as the error weight used for network training.

[0081] Error backpropagation is performed based on the network training error to update the convolutional and normalization parameters of the encoding, feature fusion, and output modules. The lightweight instance segmentation network is iteratively trained until the overlap of nozzle instance segmentation in the verification image no longer improves for several consecutive training epochs, at which point iteration stops. The number of training epochs required to stop the decision is determined based on the change in verification error; specifically, the median of the number of training epochs during which the verification error transitions from decreasing to stabilizing is taken.

[0082] The initial learning rate for network training is set based on the rate at which the validation error decreases. Specifically, pre-training is performed for the same number of training cycles under multiple candidate learning rates, and the learning rate with the fastest decrease in validation error and without continuous fluctuations is selected. The training batch size is set based on the storage capacity of the computing device and the coverage of training sample categories, ensuring that each training batch contains at least nozzle cleaning images and nozzle attachment images.

[0083] For example, the image acquisition frequency is 30 frames per second, and 16 frames are acquired for each passing of the same vehicle wheel. Historical image data includes images from the most recent 3 valid running cycles. The input image size of the lightweight instance segmentation network is 640 pixels × 640 pixels. The number of output channels of the 4-level feature extraction layers are 24, 48, 96 and 192 respectively. The semantic prediction branch outputs 4 channels, the instance center prediction branch outputs 3 channels, and the center offset prediction branch outputs 2 channels, thus obtaining the set of positioning features for the vehicle wheel nozzle.

[0084] Step S2: Based on the positioning feature set of the trolley wheel oil nozzle, perform same-wheel image matching and instance segmentation to obtain the current semantic probability map sequence, the historical semantic probability map sequence, and the instance mask sequence.

[0085] The specific steps of step S2 are as follows:

[0086] Step S201: Based on the trolley operation identification information and the trolley wheel operation images, generate a trolley identifier and arrange the trolley wheel operation images to obtain the current image sequence of the trolley wheels.

[0087] In this embodiment, based on the trolley operation identification information, the trolley number, wheel side, operation cycle, and corresponding image acquisition time range are read. The trolley number and wheel side are associated to obtain the trolley identifier. The image acquisition time of each trolley wheel operation image is read, and trolley wheel operation images whose image acquisition time falls within the specified image acquisition time range are associated with the corresponding trolley identifier and operation cycle. Trolley wheel operation images with the same trolley identifier and operation cycle are filtered and sorted in ascending order according to image acquisition time to obtain the current trolley wheel image sequence.

[0088] Step S202: Based on the historical image data and the trolley identifier, extract historical images of the same trolley wheel and perform image registration to obtain a historical passing image sequence.

[0089] In this embodiment, historical image data is retrieved based on the trolley identifier, and historical images whose trolley number and wheel side are consistent with the current trolley wheel are extracted and divided into multiple historical passing image groups according to the running cycle.

[0090] Grayscale conversion and edge enhancement are performed on the current image sequence and historical passing image group of the trolley wheel to extract corner points from the current and historical images. For each corner point, the direction and magnitude of grayscale change in its neighborhood are read and arranged according to spatial location to obtain corner point description information. The differences between the description information of the current corner point and the historical corner points are compared, and the current corner point and the historical corner point with the smallest difference in description information are formed into an initial matching point pair.

[0091] Historical images are registered using affine transformations that include translation, rotation, and proportional scaling. The affine transformations include horizontal scaling parameters, vertical scaling parameters, horizontal rotation coupling parameters, vertical rotation coupling parameters, horizontal translation parameters, and vertical translation parameters. At least three non-collinear matching point pairs are selected from the initial matching point pairs each time, and a set of candidate affine transformation parameters is calculated.

[0092] Candidate affine transform parameters are used to map historical matching points to the current image. The horizontal and vertical differences between the mapped positions and the corresponding current matching points are calculated. The squares of these two differences are summed and their square roots are taken to obtain the mapping distance between the matching points. Matching point pairs whose mapping distance is no greater than the matching point distance limit are retained. The affine transform parameters are recalculated using these retained matching point pairs until the number of retained matching point pairs no longer increases, thus obtaining the image registration parameters.

[0093] The matching point distance limit is set based on the distribution of matching point mapping distances in the verification image. Specifically, the matching point mapping distances that have been manually confirmed are sorted in ascending order. The allowable distance is gradually increased, and the registration overlap between the historical axis-end region and the current axis-end region is calculated. The minimum allowable distance corresponding to the maximum registration overlap is selected as the matching point distance limit.

[0094] Based on the image registration parameters, each historical image is mapped to the coordinate range of the current image and arranged according to the corresponding historical running cycle and image acquisition time to obtain the historical image sequence.

[0095] Step S203: Input the current image sequence and the historical passing image sequence of the trolley wheels into the lightweight instance segmentation network, perform multi-scale feature extraction and fusion, and obtain the current semantic probability map sequence, the historical semantic probability map sequence, the instance center response and the center offset.

[0096] In this embodiment, the images in the current image sequence and the historical passing image sequence of the trolley wheels are subjected to size unification and pixel normalization. During pixel normalization, the pixel mean and pixel standard deviation of each color channel of the training image are read respectively. The pixel value of each input image is subtracted from the pixel mean of the corresponding channel, and then divided by the pixel standard deviation of the corresponding channel to obtain the network input image.

[0097] The input image is fed into a lightweight instance segmentation network. The encoding module obtains encoded features at 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the input resolution through four levels of feature extraction layers. The feature fusion module upsamples the encoded features at the 1 / 160 input resolution by a factor of 2 at each level and performs channel concatenation and convolution with the encoded features at the corresponding resolutions to obtain a fused feature map.

[0098] The fused feature map corresponding to the current image is input into the semantic prediction branch to obtain the background probability, wheel body probability, axle end probability, and nozzle probability for each pixel. These probabilities are then arranged according to the image acquisition sequence to obtain the current semantic probability map sequence. Historical images are processed in the same way and arranged according to the historical operation cycle and image acquisition sequence to obtain the historical semantic probability map sequence.

[0099] The fused feature map is input into the instance center prediction branch to obtain the instance center responses corresponding to the wheel body, axle end, and nozzle. The fused feature map is input into the center offset prediction branch to obtain the lateral and vertical offsets of each foreground pixel pointing to the corresponding instance center, thus obtaining the instance center response and center offset.

[0100] Step S204: Determine the foreground pixels in the current semantic probability map sequence and the historical semantic probability map sequence, and perform instance aggregation based on the instance center response and center offset to obtain the instance mask sequence.

[0101] In this embodiment, the background probability, wheel body probability, axle end probability, and nozzle probability of each pixel in the current semantic probability map sequence and the historical semantic probability map sequence are read, and the pixels whose wheel body probability, axle end probability, or nozzle probability reaches the corresponding semantic probability limit are determined as foreground pixels.

[0102] The semantic probability constraints for each category are set based on the category recognition results in the verification image. The specific method is as follows: The category probabilities of each pixel in the verification image are arranged in ascending order of value, and different probabilities are selected sequentially as candidate semantic probability constraints. For each candidate semantic probability constraint, the number of pixels correctly identified as belonging to the corresponding category, the number of pixels incorrectly identified as belonging to the corresponding category, and the number of pixels missing from the corresponding category are counted. The precision is obtained by dividing the number of correctly identified pixels by the sum of the number of correctly identified pixels and the number of pixels missing from the corresponding category. The recall is obtained by dividing the number of correctly identified pixels by the sum of the number of correctly identified pixels and the number of pixels missing from the corresponding category. The harmonic mean is obtained by multiplying the precision and recall, multiplying by 2, and then dividing by the sum of the precision and recall. The candidate probability with the largest harmonic mean is selected as the semantic probability constraint for the corresponding category.

[0103] Based on the position and center offset of each foreground pixel, the horizontal coordinate of the foreground pixel is added to its horizontal offset, and the vertical coordinate of the foreground pixel is added to its vertical offset to obtain the predicted instance center position. Local maxima that meet the center response constraints are extracted from the instance center response to obtain candidate instance centers.

[0104] The center response limit is set based on the distribution of the center responses of real instances and false responses in the verification image. Specifically, the method is as follows: calculate the median of the center responses of real instances and the median of the center responses of false responses, add the two medians together, and divide by 2 to obtain the center response limit.

[0105] Calculate the horizontal and vertical distances between the predicted instance center and the centers of each candidate instance of the same category. Sum the squares of the two distances and take the square root to obtain the center distance. Assign the foreground pixels to the candidate instance center of the same category with the smallest center distance to obtain the wheel body instance mask, shaft end instance mask, and nozzle candidate instance mask. Arrange the instance masks according to the current image time sequence and historical running cycles to obtain the instance mask sequence.

[0106] For example, the verification image achieves the maximum harmonic mean when the semantic probability limit for the grease nipple is 0.55, and 0.55 is used as the semantic probability limit for the grease nipple. A foreground pixel has an x-coordinate of 306, a y-coordinate of 274, a lateral offset of 18, and a lateral offset of -6. Adding 306 and 18 gives the predicted instance center x-coordinate of 324, and subtracting 6 from 274 gives the predicted instance center y-coordinate of 268.

[0107] Step S3: Based on the trolley wheel running image, the current semantic probability map sequence, the historical semantic probability map sequence, and the instance mask sequence, construct the wheel and axle heterogeneous motion semantic map to obtain the oil nozzle stable structure semantic mask sequence and the common motion attachment residual sequence.

[0108] The specific steps of step S3 are as follows:

[0109] Step S301: Based on the trolley wheel running image, the current semantic probability map sequence and the instance mask sequence, extract the common translation of the axle end and perform wheel rotation phase disambiguation to obtain the axle end translation flow field and the wheel phase disambiguation rotation flow field.

[0110] The specific steps of step S301 are as follows:

[0111] Step S3011: Perform bidirectional dense optical flow estimation on the images of adjacent trolley wheels to obtain forward dense optical flow, reverse dense optical flow and bidirectional pixel correspondence confidence.

[0112] In this embodiment, based on the image acquisition time sequence, two frames of trolley wheel running images with adjacent acquisition times are grouped into a group of adjacent images, and each group of adjacent images is arranged according to the acquisition time of the previous frame image.

[0113] An image pyramid is constructed for each group of adjacent images, progressively reducing each image to half its original size. Starting from the highest level, grayscale matching is performed on each pixel in the previous frame within the corresponding search window of the next frame. The position with the smallest sum of squared neighborhood grayscale differences is selected to obtain the pixel displacement of the current level. The pixel displacement of the current level is amplified by a factor of 2 and passed to the next level. Local matching and displacement correction are performed again around the passed position until the forward dense optical flow from the previous frame to the next frame at the original resolution is obtained. The input order of the two frames is swapped, and the reverse dense optical flow is obtained through the same process.

[0114] The number of image pyramid layers is determined based on the maximum pixel displacement between adjacent images. The specific method is as follows: starting from the original image, the image is progressively scaled down, and the maximum pixel displacement of corresponding feature points in adjacent images at each layer is calculated until the maximum pixel displacement does not exceed half the width of the optical flow search window, thus obtaining the number of image pyramid layers.

[0115] The optical flow search window width is set based on the minimum visible width of the nozzle. Specifically, the minimum horizontal width of the nozzle body in the training image is statistically analyzed, and the search window width is set to be no less than this minimum horizontal width, ensuring that the search window can cover the local structure of the nozzle.

[0116] For any pixel in the previous frame, the pixel's horizontal coordinate is added to the horizontal displacement of the forward dense optical flow, and its vertical coordinate is added to the vertical displacement of the forward dense optical flow to obtain the forward mapped position. The reverse displacement corresponding to the forward mapped position is read from the reverse dense optical flow. The forward and reverse horizontal displacements are added to obtain the horizontal closure deviation, and the forward and reverse vertical displacements are added to obtain the vertical closure deviation. The squares of the horizontal and vertical closure deviations are added together, and the square root of the sum of squares is calculated to obtain the bidirectional closure error.

[0117] Calculate the median of the bidirectional closure error within the effective region of the wheel body and axle end, and then calculate the median of the absolute values ​​of the differences between each bidirectional closure error and the median, obtaining the median absolute deviation. Multiply the median absolute deviation by 1.4826 to obtain the robustness scale of the bidirectional closure error. When the robustness scale is 0, it is replaced by the minimum quantization step size of the optical flow output coordinates.

[0118] The square of the bidirectional closure error is used as the numerator of the negative exponent, and twice the square of the robust scale is used as the denominator of the negative exponent. The resulting ratio is negative and subjected to exponential transformation to obtain the bidirectional pixel correspondence confidence. The smaller the bidirectional closure error, the higher the bidirectional pixel correspondence confidence, thus obtaining the forward dense optical flow, the reverse dense optical flow, and the bidirectional pixel correspondence confidence.

[0119] Step S3012: Extract the shaft end instance region and wheel body instance region, combine the shaft end semantic probability and bidirectional pixel correspondence confidence, perform weighted aggregation and residual correction on the forward dense optical flow, and obtain the shaft end common translation amount and shaft end translation flow field.

[0120] In this embodiment, the axis-end instance mask and wheel instance mask corresponding to each group of adjacent images are read from the instance mask sequence to obtain the axis-end instance region and the wheel instance region. The axis-end semantic probability of each pixel within the axis-end instance region is read from the current semantic probability map sequence.

[0121] Multiply the x-coordinate of each pixel by its corresponding axis-end semantic probability, sum the products, and then divide by the total axis-end semantic probabilities to obtain the x-coordinate of the axis-end instance center. Multiply the y-coordinate of each pixel by its corresponding axis-end semantic probability, sum the products, and then divide by the total axis-end semantic probabilities to obtain the y-coordinate of the axis-end instance center.

[0122] The axial-end semantic probability is multiplied by the bidirectional pixel correspondence confidence to obtain the axial-end optical flow weight. The positive dense optical flows within the axial-end instance region are arranged according to the magnitude of the lateral and longitudinal displacements, respectively. The corresponding axial-end optical flow weights are accumulated, and the lateral and longitudinal displacements corresponding to when the accumulated weights reach half of the total weights are selected to obtain the initial axial-end common translation amount.

[0123] For any pixel within the axial-end instance region, subtract the initial lateral displacement from the positive lateral displacement, subtract the initial longitudinal displacement from the positive longitudinal displacement, add the squares of the two differences and perform a square root operation to obtain the axial-end optical flow residual.

[0124] The optical flow residual limit at the axial end is set based on the residual distribution within the axial end instance region. Specifically, the method is as follows: calculate the median and median absolute deviation of the optical flow residual at the axial end, and use the sum of the median and three times the median absolute deviation as the axial flow residual limit. This limit is used to exclude abnormal optical flow caused by localized reflections, occlusion, and mismatches.

[0125] For pixels whose axial optical flow residuals are no greater than the axial optical flow residual limit, the positive lateral displacement of the retained pixels is multiplied by the corresponding axial optical flow weight, summed, and then divided by the total weight of the retained pixels to obtain the lateral component of the axial common translation. The longitudinal displacement is processed in the same way to obtain the longitudinal component of the axial common translation.

[0126] The common translation amount at the axis end is assigned to the pixels within the axis end instance region. For edge pixels of the axis end instance region, the common translation amount at the axis end is multiplied by the corresponding semantic probability at the axis end to obtain the motion vector after edge smoothing, thus obtaining the axis end translation flow field.

[0127] For example, the median of the axial optical flow residual is 0.4 pixels and the median absolute deviation is 0.3 pixels. Multiplying 3 by 0.3 gives 0.9 pixels, and adding it to 0.4 pixels gives an axial optical flow residual limit of 1.3 pixels.

[0128] Step S3013: Extract the semantic boundary response and image gradient response of the wheel body within the wheel body instance region, and perform circumferential expansion and radial aggregation with the center of the axle end instance as the reference to obtain the circumferential semantic expansion sequence.

[0129] In this embodiment, the wheel semantic probability of each pixel within the wheel instance region is read from the current semantic probability map sequence. For any pixel within the wheel instance region, its wheel semantic probability is subtracted from the wheel semantic probabilities of its radially inner and radially outer adjacent pixels, respectively. The absolute values ​​of the two differences are taken and averaged to obtain the wheel semantic boundary response.

[0130] The image of the trolley wheel in motion is converted into a grayscale image. For any pixel, the grayscale value of the right-hand neighbor pixel is subtracted from the grayscale value of the left-hand neighbor pixel to obtain the horizontal grayscale difference. The grayscale value of the lower neighbor pixel is subtracted from the grayscale value of the upper neighbor pixel to obtain the vertical grayscale difference. The squares of the horizontal and vertical grayscale differences are added together and the square root is calculated. This result is then multiplied by the corresponding wheel semantic probability to obtain the image gradient response.

[0131] The circumferential sampling range is defined by using the center of the axle end instance as the circumferential expansion center and the area from the outer edge of the axle end instance region to the outer edge of the wheel body instance region. The circumferential sampling step size is set based on the perimeter of the outer edge of the wheel body instance and the minimum resolvable texture width. Specifically, the method is as follows: extract wheel body edges that can stably match in adjacent images from the verification image, calculate the lateral width of these edges, and take the minimum width as the minimum resolvable texture width. Divide the perimeter of the outer edge of the wheel body instance by the minimum resolvable texture width to obtain the number of circumferential samples, and then divide 360 ​​degrees by the number of circumferential samples to obtain the circumferential sampling step size.

[0132] In each circumferential sampling direction, the wheel semantic boundary response and image gradient response are divided by the maximum value of the corresponding response to obtain the normalized wheel semantic boundary response and normalized image gradient response. The two normalized responses are multiplied by their corresponding fusion weights and added together, and then summed along the radial sampling positions to obtain the circumferential response value.

[0133] The two fusion weights are set based on the rotation phase error in the verification image. Specifically, the wheel semantic boundary response weights are gradually changed while ensuring the sum of the two weights is 1. Following this step, a circumferential semantic unfolding sequence is generated, and the rotation phase is calculated. The weight combination with the smallest average rotation phase error is selected. The circumferential response values ​​are arranged according to the circumferential sampling position to obtain the circumferential semantic unfolding sequence corresponding to adjacent images.

[0134] Step S3014: Correct the next frame's circumferential semantic unfolding sequence based on the common translation amount of the shaft end, and perform cyclic correlation matching with the previous frame's circumferential semantic unfolding sequence to obtain the wheel rotation phase candidate and the degree of cyclic correlation.

[0135] In this embodiment, the common translation amount of each group of adjacent images is read. The horizontal component of the common translation amount is subtracted from the horizontal coordinate of the center of the axis-end instance in the next frame image, and the vertical component of the common translation amount is subtracted from the vertical coordinate to obtain the translation-corrected center of the axis-end instance. The circumferential sampling position of the next frame image is re-determined based on the translation-corrected center of the axis-end instance, resulting in the corrected circumferential semantic unfolding sequence of the next frame.

[0136] Calculate the average response values ​​of the previous frame's circumferential semantic unfolding sequence and the corrected next frame's circumferential semantic unfolding sequence, respectively. Subtract the corresponding average response value from the response value at each sampling position to obtain the mean-free sequence. Calculate the square root of the sum of squares of each response value in the mean-free sequence, and divide each mean-free response value by the corresponding square root to obtain the amplitude-normalized sequence.

[0137] The amplitude normalized sequence of the next frame is cyclically shifted to each sampling position. For any shifted position, the response values ​​at the corresponding positions in the two amplitude normalized sequences are multiplied and summed to obtain the cyclic correlation degree corresponding to the current shifted position.

[0138] Multiple local maxima are extracted from the cyclic correlation sequence to obtain candidate wheel rotation phases. The minimum interval between adjacent candidates is set based on the minimum circumferential interval of the repeating edges of the wheel. Specifically, the method is as follows: the circumferential distance between adjacent repeating edges in the verification image is statistically analyzed, the median of the circumferential distance is calculated, and then the median is divided by the outer perimeter of the wheel instance and multiplied by 360 degrees to obtain the minimum interval between adjacent candidates.

[0139] The number of candidates is set based on the correct phase coverage in the verification images. Specifically, the number of candidates is increased sequentially from high to low cyclic correlation. The number of image groups whose true rotation phase enters the candidate set is counted, and this number is divided by the total number of verification image groups to obtain the candidate coverage ratio. When the candidate coverage ratio reaches the target coverage ratio determined according to the allowable missed detection ratio for the localization task, the number of candidates is no longer increased.

[0140] For example, the circumferential sampling step size is 1 degree, and the movement positions corresponding to the local correlation peaks are 5 degrees, 35 degrees, 65 degrees and 95 degrees, respectively, with corresponding cyclic correlation degrees of 0.91, 0.86, 0.79 and 0.73, respectively. The four movement positions are determined as candidates for the wheel rotation phase.

[0141] Step S3015: Combining the forward dense optical flow, the reverse dense optical flow, and the common translation at the shaft end, the candidate rotation phase of the wheel body is verified for rotation and the continuity is screened to obtain the wheel body phase disambiguation rotational flow field.

[0142] The specific steps of step S3015 are as follows:

[0143] Existing rotation phase filtering methods typically select the candidate with the highest cyclic correlation or the candidate with the smallest residual in the forward optical flow. However, using cyclic correlation alone is insufficient to eliminate erroneous peaks caused by repetitive textures, and using the forward optical flow residual alone is susceptible to local reflections, occlusion, and mismatches. Furthermore, when adjacent images are processed independently, the rotation phase can easily jump between candidates from different cycles.

[0144] This application uses candidate motion residuals, forward and reverse rotation closure differences, and cyclic correlation to verify the first group of adjacent images, and adds rotational continuity differences to the remaining image groups, so that candidate selection is simultaneously constrained by current motion observation, bidirectional mapping reliability, cyclic structural similarity, and temporal continuity.

[0145] Step S30151: Based on the rotation phase candidates of each wheel body, rotate the pixels in the wheel body instance area around the center of the shaft end instance and superimpose the common translation amount of the shaft end to obtain the motion flow field of the wheel body candidate.

[0146] In this embodiment, for any candidate wheel rotation phase, the horizontal coordinate of each pixel in the wheel instance area is subtracted from the horizontal coordinate of the center of the shaft end instance to obtain the relative horizontal coordinate, and the vertical coordinate is subtracted from the vertical coordinate of the center of the shaft end instance to obtain the relative vertical coordinate.

[0147] Calculate the cosine and sine values ​​of the candidate angles for the wheel's rotation phase. Multiply the relative lateral coordinate by the cosine value, and the relative longitudinal coordinate by the sine value. Subtract the second product from the first to obtain the rotated relative lateral coordinate. Multiply the relative lateral coordinate by the sine value, and the relative longitudinal coordinate by the cosine value. Add the two products to obtain the rotated relative longitudinal coordinate.

[0148] The horizontal coordinates of the rotated relative coordinates are added sequentially to the horizontal coordinates of the center of the axis-end instance and the horizontal component of the common translation of the axis-end to obtain the horizontal coordinates of the candidate mapping positions. The vertical coordinates of the rotated relative coordinates are added sequentially to the vertical coordinates of the center of the axis-end instance and the vertical component of the common translation of the axis-end to obtain the vertical coordinates of the candidate mapping positions.

[0149] Subtracting the original pixel's x-coordinate from the candidate mapping position's x-coordinate yields the horizontal component of the candidate motion vector; subtracting the original pixel's y-coordinate from the candidate mapping position's y-coordinate yields the vertical component of the candidate motion vector. Summarizing the candidate motion vectors of each pixel within the wheel instance region yields the candidate motion flow field of the wheel.

[0150] Conventional rigid body rotation only describes the rotational displacement of the wheel body relative to the axle end, excluding the overall translation of the trolley. This application superimposes the common translation of the axle end in the rotational mapping, so that the candidate motion flow field of the wheel body simultaneously represents the wheel body rotation and the trolley translation, avoiding the miscalculation of the common translation of the axle end as the candidate rotation error.

[0151] For example, the x-coordinate and y-coordinate of the center of the axis end instance are both 320 pixels. The x-coordinate of a certain wheel body pixel is 420 pixels, the y-coordinate is 320 pixels, the candidate rotation phase is 6 degrees, and the horizontal component of the common translation of the axis end is 5 pixels, and the vertical component is 1 pixel. The relative horizontal coordinate of this pixel is 100 pixels, and the relative vertical coordinate is 0 pixels. The cosine value of 6 degrees is approximately 0.9945, and the sine value is approximately 0.1045. Multiplying 100 by 0.9945 gives a rotated relative horizontal coordinate of approximately 99.45 pixels, and multiplying 100 by 0.1045 gives a rotated relative vertical coordinate of approximately 10.45 pixels. Adding 99.45, 320, and 5 gives a candidate mapping position x-coordinate of 424.45 pixels, and adding 10.45, 320, and 1 gives a candidate mapping position y-coordinate of 331.45 pixels. Subtracting 420 from 424.45 yields a horizontal component of 4.45 pixels for the candidate motion vector, and subtracting 320 from 331.45 yields a vertical component of 11.45 pixels for the candidate motion vector.

[0152] Step S30152: Compare the candidate motion flow field of the wheel body with the forward dense optical flow to obtain the candidate motion residual, and use the reverse dense optical flow to perform reverse mapping to obtain the forward and reverse rotational closure difference.

[0153] In this embodiment, for any candidate wheel rotation phase and any pixel within the wheel instance region, the lateral displacement of the positive dense optical flow is subtracted from the lateral displacement of the candidate wheel motion flow field to obtain the lateral motion difference. Similarly, the longitudinal displacement of the positive dense optical flow is subtracted from the longitudinal displacement of the candidate wheel motion flow field to obtain the longitudinal motion difference. The squares of the lateral and longitudinal motion differences are added together and the square root is calculated to obtain the pixel candidate motion residual.

[0154] Multiply the pixel candidate motion residual, the bidirectional pixel correspondence confidence, and the wheel semantic probability, and sum the products over the wheel instance region. Multiply the bidirectional pixel correspondence confidence and the wheel semantic probability and sum them. Divide the sum of the weighted products of the aforementioned residuals by the sum of the weights described below to obtain the candidate motion residual corresponding to the current candidate.

[0155] Based on the candidate motion flow field of the wheel, each pixel within the wheel instance region is mapped to the next frame image, and the reverse dense optical flow corresponding to the mapped position is read. The horizontal coordinate of the mapped position is added to the reverse horizontal displacement, and the vertical coordinate of the mapped position is added to the reverse vertical displacement to obtain the reverse mapped position.

[0156] Subtracting the original pixel's x-coordinate from the reverse-mapped position's x-coordinate yields the horizontal closure difference; subtracting the original pixel's y-coordinate from the reverse-mapped position's y-coordinate yields the vertical closure difference. Summing the squares of the horizontal and vertical closure differences and taking the square root gives the pixel's forward and reverse rotation closure difference. A weighted average of these differences is then applied based on the bidirectional pixel correspondence confidence and the wheel semantic probability to obtain the forward and reverse rotation closure difference corresponding to the current candidate pixel.

[0157] Conventional unidirectional optical flow residuals cannot determine whether mapped pixels can return to their original positions. This application adds forward and reverse rotation closure differences, so that even if erroneous candidates obtain small forward residuals in a short time, they will incur high costs due to the inability to close the bidirectional mapping.

[0158] For example, the candidate motion vector for a certain pixel is 4.45 pixels horizontally and 11.45 pixels vertically, while the forward dense optical flow is 4.80 pixels horizontally and 11.00 pixels vertically. Subtracting 4.45 from 4.80 yields a horizontal motion difference of 0.35 pixels, and subtracting 11.45 from 11.00 yields a vertical motion difference of -0.45 pixels. Adding the square of 0.35 to the square of -0.45 gives 0.325. Taking the square root of 0.325 yields a candidate motion residual of approximately 0.57 pixels. This calculation result is correct.

[0159] Step S30153: Select the first group of target rotation phases based on the candidate motion residuals, the difference between forward and reverse rotation closures, and the degree of cyclic correlation.

[0160] In this embodiment, adjacent images are arranged according to the acquisition time of the previous frame image, and the group with the earliest acquisition time is determined as the first group of adjacent images.

[0161] Divide the candidate motion residuals of each candidate in the first group by the maximum candidate motion residual to obtain the normalized candidate motion residuals. Divide the forward and reverse rotation closure differences by the maximum closure difference to obtain the normalized closure difference. Divide the cyclic correlation degree by the maximum cyclic correlation degree, and then subtract the result from 1 to obtain the cyclic correlation cost.

[0162] The normalized candidate motion residuals, normalized closure differences, and cyclic correlation costs are multiplied by their corresponding weights, and then the three products are summed to obtain the first set of screening costs.

[0163] The three weights are set based on the average rotation phase error of the verification images. Specifically, the weight combinations are iterated while ensuring the sum of the three weights equals 1. The absolute value of the angular difference between the target rotation phase and the labeled rotation phase is calculated for each weight combination, and then averaged over the verification image set to obtain the average rotation phase error. The weight combination with the smallest average rotation phase error is selected. The candidate with the lowest screening cost in the first group is determined as the first group of target rotation phases.

[0164] For example, the normalized candidate motion residuals of the three candidates are 0.20, 0.55, and 0.80, respectively; the normalized closure difference is 0.25, 0.50, and 0.75, respectively; the cycle correlation cost is 0.10, 0.18, and 0.25, respectively; and the three weights are 0.45, 0.35, and 0.20, respectively. The screening cost of the first candidate is the sum of the products of 0.45 and 0.20, 0.35 and 0.25, and 0.20 and 0.10, i.e., 0.09 plus 0.0875 plus 0.02, which equals 0.1975, approximately 0.20. The screening cost of the second candidate is 0.2475 plus 0.175 plus 0.036, which equals 0.4585, approximately 0.46. The screening cost for the third candidate is 0.36 + 0.2625 + 0.05, which equals 0.6725, approximately 0.67. All the calculation results are correct, therefore the first candidate is determined as the target rotation phase.

[0165] Step S30154: According to the image order, and combining the continuity difference between the current candidate and the previous group of target rotation phases, filter the remaining target rotation phases.

[0166] In this embodiment, for any candidate wheel rotation phase in the current group, the target rotation phase of the previous group is subtracted from the current candidate rotation phase to obtain an initial angle difference. When the initial angle difference is greater than 180 degrees, 360 degrees is subtracted from the initial angle difference. When the initial angle difference is less than -180 degrees, 360 degrees is added to the initial angle difference. The absolute value of the adjusted angle difference is taken to obtain the rotational continuity difference.

[0167] The candidate motion residuals, forward and reverse rotation closure differences, and rotational continuity differences are each divided by their respective maximum values ​​in the current candidate set to obtain the normalized candidate motion residuals, normalized closure differences, and normalized rotational continuity differences. The cyclic correlation cost is obtained by dividing the degree of cyclic correlation by the maximum degree of cyclic correlation and then subtracting the result from 1.

[0168] The normalized candidate motion residual, normalized closure difference, cyclic correlation cost, and normalized rotational continuity difference are multiplied by their respective weights, and then the four products are summed to obtain the continuous screening cost.

[0169] The weight of rotational continuity difference is set based on the actual rotational phase change of adjacent groups in the verification image. Specifically, the weight is increased gradually while ensuring that no actual phase candidate is excluded. The standard deviation of the target rotational phase change of adjacent image groups is calculated, and the weight with the smallest standard deviation is selected. The candidate with the lowest continuous screening cost is determined as the target rotational phase of the current group, and the target rotational phases of the remaining groups are obtained sequentially.

[0170] Step S30155: Based on the target rotation phase and the common translation amount of the shaft end of each group, perform rotation and translation mapping on the wheel body instance region to obtain the wheel body phase disambiguation rotational flow field.

[0171] In this embodiment, the target rotation phase is summarized according to the arrangement order of adjacent image groups. For the wheel instance region in each group of adjacent images, the coordinates corresponding to the center of the axis end instance are subtracted from the x and y coordinates of each pixel to obtain the relative coordinates. The relative coordinates are rotated according to the cosine and sine values ​​of the target rotation phase, and then the rotated coordinates are added to the center of the axis end instance and the common translation of the axis end to obtain the target mapping position.

[0172] Subtracting the original pixel's x-coordinate from the target's mapped position's x-coordinate yields the horizontal component of the target's motion vector; subtracting the original pixel's y-coordinate from the target's mapped position's y-coordinate yields the vertical component of the target's motion vector. Weighted smoothing of the target motion vectors at the wheel instance region edges is then performed based on the wheel's semantic probability, and the vectors are arranged according to image temporal sequence to obtain the wheel's phase-disambiguated rotating flow field.

[0173] Step S302: Based on the trolley operation identification information, the current semantic probability map sequence, the historical semantic probability map sequence, the instance mask sequence, and the flow field, construct the wheel-axle heterogeneous motion semantic map.

[0174] The specific steps of step S302 are as follows:

[0175] Step S3021: Extract historical and current shaft end instances, wheel body instances, and oil nozzle candidate instances from the current semantic probability graph sequence, historical semantic probability graph sequence, and instance mask sequence, establish corresponding nodes, and obtain a heterogeneous semantic node set.

[0176] In this embodiment, the axle end instance mask, wheel body instance mask, and nozzle candidate instance mask corresponding to the current image and the historical passed image are read from the instance mask sequence to obtain the current axle end instance, historical axle end instance, current wheel body instance, historical wheel body instance, and nozzle candidate instance.

[0177] Nodes are created for each axle end instance, wheel body instance, and nozzle candidate instance. Each node records the trolley identifier, operating cycle, image acquisition time sequence, instance center, instance area, and average semantic probability. The instance center is the average of the coordinates of each pixel within the instance mask. The instance area is the number of pixels within the instance mask. The average semantic probability is the sum of the semantic probabilities of the corresponding category within the instance mask divided by the number of instance pixels. The nodes are arranged according to the trolley identifier, operating cycle, and image acquisition time sequence to obtain a heterogeneous semantic node set.

[0178] Step S3022: Using the shaft end translation flow field to map historical shaft end instances, establish shaft end translation receiving edges based on the degree of overlap and positional deviation between the mapped region and the current shaft end instance.

[0179] In this embodiment, the historical image sequence has been registered to the current image coordinate range. Based on the relative acquisition timing of historical axis-end instances in the historical image sequence, the axis-end translational flow field corresponding to the same relative acquisition timing in the current cycle is read, and the axis-end translational flow field is used as the motion template for the current cycle.

[0180] Add the x-coordinate of each pixel within the historical axis-end instance to the lateral displacement of the axis-end translational flow field, and add the y-coordinate to the longitudinal displacement of the axis-end translational flow field to obtain the mapped region of the historical axis-end instance.

[0181] The number of intersection pixels and the number of union pixels between the mapped region and the current axis-end instance are counted. The overlap is obtained by dividing the number of intersection pixels by the number of union pixels. The horizontal and vertical differences between the center of the mapped region and the center of the current axis-end instance are calculated separately. The squares of the two differences are added together and the square root is calculated to obtain the positional deviation.

[0182] Divide the area of ​​the current pivot instance by pi, and take the square root of the result to obtain the equivalent radius of the current pivot instance. Divide the positional deviation by the equivalent radius of the current pivot instance to obtain the normalized positional deviation. Negate the normalized positional deviation and perform an exponential transformation, then multiply it by the degree of overlap to obtain the pivot translation acceptance degree. Establish pivot translation acceptance edges between the corresponding historical pivot nodes and the current pivot node, and use the pivot translation acceptance degree as the edge weight.

[0183] Step S3023: Using the wheel phase disambiguation rotating flow field mapping of historical wheel instances, establish wheel rotation bearing edges based on the degree of overlap and semantic distribution differences between the mapped region and the current wheel instance.

[0184] In this embodiment, based on the relative acquisition time sequence of historical wheel instances in the historical image sequence, the wheel phase disambiguation rotating flow field corresponding to the same relative acquisition time sequence in the current period is read, and the position of each pixel in the historical wheel instance is added to the corresponding motion vector to obtain the mapping region.

[0185] The number of intersection pixels and the number of union pixels between the mapped region and the current wheel instance are counted. The number of intersection pixels is divided by the number of union pixels to obtain the degree of overlap of the wheel mapping.

[0186] The historical wheel semantic probabilities are mapped to the current image based on the wheel phase-disambiguated rotating flow field. The probability range of 0 to 1 is divided into multiple continuous probability intervals. The number of pixels whose mapped historical and current wheel semantic probabilities fall into each probability interval is counted. Then, each interval is divided by the total number of pixels of the corresponding wheel instance to obtain the pixel proportion of each probability interval.

[0187] The number of probability intervals is set based on the stability of semantic distribution differences. The specific setting method is as follows: starting with a small number of probability intervals, gradually increase by one interval, calculate the change in semantic distribution differences obtained from two adjacent interval divisions, and stop increasing when the change no longer decreases after continuing to increase the number of probability intervals. The corresponding number of intervals is then used as the number of intervals adopted.

[0188] Calculate the absolute value of the difference between the percentage of pixels in the corresponding probability intervals of the historical wheel semantic probability and the current wheel semantic probability after mapping. Sum all the absolute values ​​of the differences and divide by 2 to obtain the normalized semantic distribution difference. The value range of the normalized semantic distribution difference is 0 to 1. Subtract the normalized semantic distribution difference from 1 to obtain the semantic distribution consistency.

[0189] The wheel mapping overlap degree and semantic distribution consistency degree are multiplied by their corresponding weights and then summed to obtain the wheel rotation continuity degree. These two weights are set based on the accuracy of wheel instance correspondence in the verification image; specifically, the weights are adjusted when the sum of the two weights is 1, selecting the weight combination with the highest proportion of correctly establishing wheel instance correspondences. Wheel rotation continuity edges are then established between historical wheel nodes and the current wheel node.

[0190] Step S3024: Extract the historical connection root region and current connection root region of the same trolley wheel based on the trolley operation identification information, and establish the connection root reproduction edge by aligning the axle end reference and comparing the reproduction.

[0191] The specific steps of step S3024 are as follows:

[0192] Step S30241: Extract the current connection root region based on the connection position between the nozzle candidate instance mask and the shaft end instance mask.

[0193] In this embodiment, the current nozzle candidate instance mask and the current axis-end instance mask are read from the instance mask sequence, and their boundaries are extracted respectively. For any pixel within the current nozzle candidate instance mask, the horizontal difference and vertical difference between the pixel and each axis-end boundary pixel are calculated respectively. The squares of the two differences are added together and the square root is calculated. The minimum distance is selected as the axis-end boundary distance of the pixel.

[0194] Divide the area of ​​the shaft end instance by pi, take the square root of the result, and multiply by 2 to obtain the equivalent diameter of the shaft end. Multiply the equivalent diameter of the shaft end by the ratio of the connection root distance to obtain the connection root distance limit.

[0195] The connector root distance ratio is set based on the actual connector root positions in the labeled image. The specific method is as follows: For each verification image, calculate the distance from the axis-end boundary of the farthest pixel in the actual connector root region, then divide this distance by the corresponding equivalent axis-end diameter to obtain the connector root distance ratio. The maximum value among all manually verified valid connector root distance ratios is taken as the connector root distance ratio.

[0196] Candidate pixels for the nozzle that are no more than the distance from the axis end boundary to the connection root are identified as candidate pixels for the connection root. Connected region aggregation is performed on the candidate pixels for the connection root, and the connected region with the largest area that is connected to the main body of the candidate nozzle instance is identified as the current connection root region.

[0197] Step S30242: Based on the trolley operation identification information, extract the historical oil nozzle candidate instance mask and historical axle end instance mask of the same trolley wheel from the historical semantic probability map sequence and instance mask sequence to determine the historical connection root region.

[0198] In this embodiment, the historical nozzle candidate instance mask and historical axle end instance mask of the same vehicle wheel are extracted based on the vehicle number and wheel side. Following the distance calculation process in step S30241, the minimum distance from the historical nozzle candidate instance pixel to the historical axle end instance boundary is calculated.

[0199] Calculate the historical axis-end equivalent diameter, and multiply the historical axis-end equivalent diameter by the connection root distance ratio determined in step S30241 to obtain the historical connection root distance limit. Collect connected regions from historical nozzle candidate pixels whose minimum distance is not greater than the historical connection root distance limit, and retain the connected region with the largest area that is connected to the main body of the historical nozzle candidate instance to obtain the historical connection root region.

[0200] Step S30243: Determine the current axis-end geometric reference and the historical axis-end geometric reference based on the axis-end center, region size and main axis direction of the current axis-end instance mask and the historical axis-end instance mask, respectively.

[0201] In this embodiment, the abscissas of all pixels within the current axis-end instance mask are summed, and then divided by the number of pixels in the current axis-end instance to obtain the abscissa of the current axis-end center. The ordinates are processed in the same way to obtain the ordinate of the current axis-end center.

[0202] Count the number of pixels in the current axis-end instance mask to obtain the area of ​​the current axis-end region. Divide the area of ​​the current axis-end region by pi, take the square root of the result, and multiply by 2 to obtain the size of the current axis-end region.

[0203] Subtract the x-coordinate of the current axis-end center from the x-coordinate of each pixel within the current axis-end instance to obtain the lateral coordinate difference. Subtract the y-coordinate of the current axis-end center from the y-coordinate to obtain the lateral coordinate difference. Calculate the average of the squares of the lateral and lateral coordinate differences, as well as the average of the products of the lateral and lateral coordinate differences, to construct a 2x2 coordinate covariance matrix. Perform eigenvalue decomposition on the coordinate covariance matrix, and determine the direction corresponding to the largest eigenvalue as the current axis-end principal axis direction to obtain the current axis-end geometric reference. Process the historical axis-end instance masks in the same way to obtain the historical axis-end geometric references.

[0204] Step S30244: Construct the current axis-end local coordinate system based on the geometric datum, and map the historical connection root region to the local coordinate system to obtain the aligned historical connection root region.

[0205] In this embodiment, a local coordinate system for the current axis end is constructed by taking the center of the current axis end as the origin, the main axis direction of the current axis end as the first coordinate axis direction, and the direction perpendicular to the main axis direction of the current axis end as the second coordinate axis direction.

[0206] Divide the current axis-end region size by the historical axis-end region size to obtain the size scaling ratio. Subtract the historical axis-end spindle direction angle from the current axis-end spindle direction angle to obtain the direction rotation amount.

[0207] For any pixel within the historical connection root region, subtract the historical axis center coordinate from the pixel's x-coordinate and subtract the historical axis center coordinate from the pixel's y-coordinate to obtain the historical axis relative coordinate. Multiply the historical axis relative coordinate by the size scaling ratio to obtain the scaling relative coordinate.

[0208] The scaling relative coordinates are rotated based on the cosine and sine values ​​of the directional rotation. The rotated horizontal coordinate is then added to the current axis-end center horizontal coordinate, and the rotated vertical coordinate is added to the current axis-end center vertical coordinate to obtain the mapped position. The mapped positions of each pixel within the historical connection root region are then summarized to obtain the aligned historical connection root region.

[0209] Step S30245: Compare the current connection root region with the aligned historical connection root region to determine the degree of connection root repetition and establish connection root repetition edges.

[0210] In this embodiment, the number of intersection pixels and the number of union pixels between the current connection root region and the aligned historical connection root region are counted. The number of intersection pixels is divided by the number of union pixels to obtain the degree of region overlap.

[0211] Calculate the lateral and longitudinal differences between the centers of the two connecting root regions, add the squares of the two differences, and take the square root to obtain the center position distance. Divide the center position distance by the current axis end region size to obtain the normalized center position difference. Take the negative value of the difference and perform an exponential transformation to obtain the center position consistency.

[0212] Construct a direction vector pointing from the current axis center to the current connection root center, and a direction vector pointing from the current axis center to the alignment with the historical connection root center. Divide the inner product of the two direction vectors by the product of their lengths to obtain the direction cosine value, then perform an inverse cosine operation to obtain the direction angle. Divide the direction angle by 180 degrees, and then subtract the result from 1 to obtain the degree of consistency of the connection directions.

[0213] The semantic probabilities of the nozzles in the current connection root region and the aligned historical connection root region are divided into the same probability intervals as in step S3023, and the pixel proportion of each interval is calculated. The absolute value of the difference in pixel proportions between corresponding intervals is calculated, and all absolute values ​​are summed and divided by 2 to obtain the normalized semantic distribution difference. The normalized semantic distribution difference is subtracted from 1 to obtain the semantic distribution consistency.

[0214] The degree of single-time recurrence is obtained by multiplying the degree of regional overlap, the degree of consistency of center position, the degree of consistency of connection direction, and the degree of consistency of semantic distribution by their respective weights and summing them. The degree of single-time recurrence for each historical running cycle of the same vehicle wheel is then summed and divided by the number of historical running cycles to obtain the degree of recurrence of the connecting root.

[0215] The weights are set based on the distinction between true connected roots and non-connected roots in the labeled image. The specific setting method is as follows: adjust the weights with the sum of all weights equal to 1, calculate the average recurrence of true connected roots and the average recurrence of non-connected roots respectively, subtract the latter from the former to obtain the distinction difference, and select the weight combination with the largest distinction difference.

[0216] Establish a root recurrence edge between the candidate node of the nozzle and the corresponding shaft end node, and use the degree of root recurrence as the edge weight.

[0217] For example, the region overlap is 0.82, the center location consistency is 0.90, the connection direction consistency is 0.88, and the semantic distribution consistency is 0.86, with each of the four weights being 0.25. Multiplying each of the four consistency values ​​by 0.25 and summing the results, we get 0.205 + 0.225 + 0.22 + 0.215, which equals 0.865. This calculation result is correct.

[0218] Step S3025: Organize the heterogeneous semantic node set and various edges into a graph structure to obtain the wheel-axle heterogeneous motion semantic graph.

[0219] In this embodiment, the heterogeneous semantic node set is used as the graph node set, and the axle-end translational receiving edge, wheel rotational receiving edge, and connection root reproduction edge are used as the graph edge set. Historical axle-end nodes and current axle-end nodes are connected via axle-end translational receiving edges, historical wheel nodes and current wheel nodes are connected via wheel rotational receiving edges, and nozzle candidate nodes and their corresponding axle-end nodes are connected via connection root reproduction edges. Nodes and graph edges are indexed according to trolley identification, operating cycle, and image acquisition sequence to construct a heterogeneous motion semantic graph of wheels and axles.

[0220] Step S303: Based on the heterogeneous motion semantic map of the wheel and axle and the flow field, determine the candidate pixels of the nozzle, construct the root continuation semantic clique and the rotation traction semantic clique, and obtain the high-order conditional random field.

[0221] The specific steps of step S303 are as follows:

[0222] Step S3031: Determine the candidate pixels of the nozzle from the current semantic probability map sequence and the instance mask sequence, extract the semantic probabilities of the nozzle, shaft end and wheel body, and obtain the pixel set to be inferred.

[0223] In this embodiment, the current nozzle candidate instance mask is read from the instance mask sequence, and the nozzle semantic probability, shaft end semantic probability, wheel body semantic probability and background semantic probability of each pixel within the mask coverage area are read from the current semantic probability map sequence.

[0224] The candidate pixel probability limit is set based on the retention ratio of stable nozzle structure pixels and the introduction ratio of non-nozzle pixels in the verification image. Specifically, the method is as follows: different semantic probabilities of nozzles appearing within the candidate nozzle instance range in the verification image are sequentially used as candidate limits. For each candidate limit, the number of pixels in the stable nozzle structure pixels that meet the candidate limit is counted, and this number is divided by the total number of stable nozzle structure pixels to obtain the stable structure retention ratio. The number of pixels in the non-nozzle pixels that meet the candidate limit is counted, and this number is divided by the total number of non-nozzle pixels to obtain the non-nozzle pixel introduction ratio. The candidate limit with the lowest stable structure retention ratio that satisfies the allowable omission ratio for the localization task and the lowest non-nozzle pixel introduction ratio is selected.

[0225] Pixels whose semantic probability of the nozzle reaches the candidate pixel probability limit are identified as first candidate pixels. Pixels directly adjacent to the first candidate pixels, whose semantic probability of the axle end or wheel body is higher than the background semantic probability, are identified as supplementary candidate pixels. The first candidate pixels and supplementary candidate pixels are summarized, and each pixel and its semantic probability are arranged according to the image acquisition time sequence to obtain the pixel set to be inferred.

[0226] For example, verification was performed with a test interval of 0.05 in the candidate range of 0.20 to 0.70. When the candidate pixel probability limit was 0.40, the retention ratio of the oil nozzle stable structure pixel was 0.97, and the introduction ratio of non-oil nozzle pixels was the smallest. 0.40 was used as the candidate pixel probability limit.

[0227] Step S3032: Motion mapping is performed on the pixel to be inferred using the shaft end translational flow field and the wheel phase disambiguation rotating flow field to construct the single pixel potential and pixel pair potential.

[0228] In this embodiment, four candidate labels are set for each pixel to be inferred: stable nozzle, co-motion attachment, wheel rotation traction, and background. The stable nozzle label represents the metal body structure of the nozzle, the co-motion attachment label represents grease or dust that moves with the shaft end but does not belong to the nozzle body, the wheel rotation traction label represents pixels that are missegmented as the wheel rotates, and the background label represents pixels that do not belong to the aforementioned areas.

[0229] Read the forward dense optical flow obtained in step S3011. For any pixel to be inferred, subtract the lateral component of the axis-end translational flow field from the lateral component of the forward dense optical flow, and subtract the longitudinal component of the axis-end translational flow field from the longitudinal component. Add the squares of the two differences and perform a square root operation to obtain the axis-end motion deviation.

[0230] Subtract the transverse component of the wheel phase-disambiguated rotating flow field from the transverse component of the positive dense optical flow, and subtract the longitudinal component of the wheel phase-disambiguated rotating flow field from the longitudinal component. Add the squares of the two differences and take the square root to obtain the wheel rotation deviation.

[0231] The median and median absolute deviation of the axis-end motion deviation within the axis-end instance region are calculated separately. The sum of the median and median absolute deviation is used as the axis-end motion deviation normalization scale. The axis-end motion deviation of the pixel to be inferred is divided by the scale, and any result greater than 1 is cropped to 1 to obtain the normalized axis-end motion deviation. In the same manner, the wheel rotation deviation normalization scale is determined based on the wheel rotation deviation within the wheel instance region to obtain the normalized wheel rotation deviation.

[0232] The single-pixel potential of the stable nozzle label is obtained by multiplying 1 by the semantic probability of the nozzle, the normalized axle end motion deviation, and 1 by the normalized wheel rotation deviation, each with its corresponding weight, and then summing the results. The single-pixel potential of the co-movement attached label is obtained by multiplying 1 by the semantic probability of the nozzle, the normalized axle end motion deviation, and 1 by the normalized axle end motion deviation, each with its corresponding weight, and then summing the results. The single-pixel potential of the wheel rotation traction label is obtained by multiplying 1 by the semantic probability of the nozzle, the normalized wheel rotation deviation, and 1 by the normalized axle end motion deviation, each with its corresponding weight, and then summing the results. The single-pixel potential of the background label is obtained by multiplying the semantic probability of the nozzle, the semantic probability of the axle end, and the semantic probability of the wheel body, each with its corresponding weight, and then summing the results.

[0233] The weights of each term in the single-pixel potential are set according to the numerical scale of each energy term in the verification image. The specific setting method is as follows: the median of each energy term is calculated, the reciprocal of each median is normalized to the initial weight, and then adjusted within the range of 0.5 to 2 times the initial weight, selecting the weight combination with the highest degree of overlap in the segmentation of the nozzle stable structure.

[0234] For two spatially adjacent pixels to be inferred, calculate their image location distance, grayscale difference, semantic probability difference, and forward optical flow difference. The normalization scale for each difference is set based on the median of the corresponding difference in the validation image. Divide each difference by the corresponding normalization scale, square the result, take the negative value, and perform an exponential transformation. Then, sum the transformed results in a weighted manner to obtain the pixel similarity. When adjacent pixels have different labels, the pixel similarity is used as the pixel pair potential; when adjacent pixels have the same label, the pixel pair potential is set to 0.

[0235] For example, the semantic probability of a certain pixel's nozzle is 0.82, the normalized axis-end motion deviation is 0.12, the normalized wheel rotation deviation is 0.71, and the weights of the three factors for stabilizing the nozzle label are all one-third. Subtracting 0.82 from 1 gives 0.18, and subtracting 0.71 from 1 gives 0.29. Adding 0.18, 0.12, and 0.29 gives 0.59, and dividing by 3 gives a single-pixel potential of approximately 0.1967 for stabilizing the nozzle label, which is approximately 0.20. This calculation result is correct.

[0236] like Figure 2 As shown, step S3033: Determine the association between the pixel to be inferred and the axle end instance based on the wheel-axle heterogeneous motion semantic graph, collect the axle end structure continuation pixels, and obtain the root continuation semantic group.

[0237] In this embodiment, the actual nozzle body forms a stable connection with the axle end through a connecting root, and the connecting root can be reproduced in multiple operating cycles of the same vehicle wheel. Based on this feature, the connecting root reproduction edges between the nozzle candidate node and the axle end node are read from the wheel-axle heterogeneous motion semantic graph, and the current connecting root region and the degree of connecting root reproduction are read.

[0238] The pixel to be inferred within the current root region is determined as the root starting pixel, and the search proceeds outward along the candidate instance mask for the nozzle from the root starting pixel. For any adjacent pixel, the nozzle semantic probability is subtracted from 1 to obtain the missing nozzle semantic value. The normalized axis-end motion deviation of the pixel is read. The absolute value of the difference between the nozzle semantic probabilities of the current pixel and its adjacent pixels is calculated to obtain the semantic change.

[0239] The pixel splicing cost is obtained by multiplying the missing semantic value of the nozzle, the normalized shaft end motion deviation, and the semantic change by their respective weights and summing the results. The weights of the pixel splicing cost are set according to the degree of distinction between the real nozzle main path and the attached interference path. Specifically, the weights are adjusted under the condition that the sum of the weights is 1, and the average splicing cost of the two types of paths is calculated separately. The average splicing cost of the attached interference path is subtracted from the average splicing cost of the real nozzle main path to obtain the path distinction difference. The weight combination with the largest path distinction difference is selected.

[0240] Multiple candidate continuation paths are constructed, extending from the root starting pixel to the outer edge of the nozzle candidate instance. The continuation costs of all pixels in each path are summed to obtain the path continuation cost. Candidate paths are retained in ascending order of path continuation cost.

[0241] The stopping condition for candidate path retention is set based on the width of the actual nozzle body in the verification image. Specifically, the minimum width of the actual nozzle body along the vertical principal axis in the verification image is statistically analyzed. When the horizontal width covered by the retained path and its adjacent pixels reaches the minimum width, the addition of candidate paths stops. The retained paths and their directly adjacent pixels are aggregated, and the recurrence degree of the connecting root is used as the semantic cluster confidence weight to obtain the root continuation semantic cluster.

[0242] exist Figure 2 In the diagram, the three dashed boxes, from left to right, represent the determination of the nozzle candidate region, the screening of candidate continuation paths, and the formation of the root continuation semantic group, respectively. The arrows between the dashed boxes indicate the progressive relationship between each processing stage.

[0243] Within the dashed box on the left, the shaft end instance represents the shaft end structure connected to the nozzle, the diagonally filled area represents the current connection root region where the nozzle candidate instance and the shaft end instance are connected, the irregular dashed boundary represents the nozzle candidate region, the hollow square represents the pixels to be inferred distributed within the nozzle candidate region, and the dotted filled area represents the attachment interference region connected to the nozzle body.

[0244] Within the central dashed box, circular nodes located at the edge of the current root region and connected to each path represent the root starting pixel. Hollow circular nodes and dashed paths connecting them represent candidate continuation paths extending from the root starting pixel outwards from the nozzle candidate region. Dotted filled areas represent attachment interference regions that may be traversed during the extension of the candidate paths. Based on the nozzle semantic missing amount, shaft end motion deviation, and semantic change between adjacent pixels, the continuation cost of each pixel is determined. The continuation costs of pixels within the same path are accumulated to obtain the path continuation cost. Solid circular nodes and solid paths connecting them represent candidate continuation paths retained after path continuation cost comparison; the remaining dashed paths represent candidate continuation paths that were not retained.

[0245] In the dashed box on the right, the squares arranged continuously along the main body of the nozzle represent the pixel set obtained by collecting the preserved path and its directly adjacent pixels to be inferred. The outer dotted outline represents the coverage of the root continuation semantic group. The horizontal arrow indicates the extension along the main body of the nozzle, thus obtaining the root continuation semantic group that maintains a structural continuation relationship with the shaft end instance.

[0246] Step S3034: Determine the association between the pixel to be inferred and the wheel instance based on the heterogeneous motion semantic map of the wheel axle and the phase disambiguation rotating flow field of the wheel body, collect the continuous pixels of the rotation mapping, and obtain the rotation traction semantic group.

[0247] In this embodiment, the application reads the wheel rotation bearing edge and its weight from the heterogeneous motion semantic map of the wheel axle, and reads the wheel phase disambiguation rotational flow field. For any pixel to be inferred, its position is added to the corresponding motion vector of the wheel phase disambiguation rotational flow field to obtain the wheel rotation mapping position. The horizontal and vertical differences between the mapping position and each pixel to be inferred in the next frame are calculated respectively. The squares of the two differences are added together and the square root is calculated to obtain the distance of the mapping position.

[0248] The rotation mapping distance limit is set based on the forward and reverse rotation closure differences within the wheel instance region. The specific setting method is as follows: calculate the median and median absolute deviation of the forward and reverse rotation closure differences, multiply the median absolute deviation by 1.4826 to obtain the rotation closure difference robustness scale, and use the sum of the median and 3 times the robustness scale as the rotation mapping distance limit.

[0249] Establish rotation mapping relationships for pixels whose mapped positions are no greater than the rotation mapping distance limit. Connect the pixels with rotation mapping relationships along the image acquisition time sequence to obtain candidate rotation trajectories.

[0250] For any candidate rotation trajectory, sum the wheel rotation deviations of each pixel within the trajectory, and then divide by the number of pixels in the trajectory to obtain the average wheel rotation deviation. Sum the axis motion deviations of each pixel within the trajectory, and then divide by the number of pixels in the trajectory to obtain the average axis motion deviation.

[0251] Candidate rotation trajectories with average wheel rotation deviation less than average axle end motion deviation are collected, and the edge weights of the corresponding wheel rotation bearing edges are used as trajectory confidence weights to obtain rotation traction semantic groups.

[0252] Step S3035: Construct a connection root continuation potential and a wheel rotation repulsion potential based on two types of semantic cliques, and combine them with single-pixel potential and pixel pair potential to obtain a higher-order conditional random field.

[0253] In this embodiment, the application constructs a connection root continuation potential based on the root continuation semantic group and a wheel rotation repulsion potential based on the rotation traction semantic group. The connection root continuation potential encourages stable nozzle labels to extend continuously from the connection root along the nozzle body, but does not restrict attached pixels outside the path. The wheel rotation repulsion potential increases the cost of using stable nozzle labels for pixels in the rotation traction semantic group, but does not use hard rules to directly delete pixels.

[0254] For any contiguous semantic cluster, arrange the pixels to be inferred in order from the connecting root region to the outer edge of the candidate nozzle instance. Count the number of pixels with stable nozzle labels, divide the number by the total number of pixels in the semantic cluster to obtain the percentage of pixels with stable nozzle labels. Subtract the percentage of pixels with stable nozzle labels from 1 to obtain the degree of label missing.

[0255] The number of times a stable nozzle label changes to other labels is counted in the order of arrangement to obtain the label interruption count. The label interruption count is divided by the total number of semantic cluster pixels to obtain the normalized interruption degree. The label missing degree is multiplied by itself to obtain the label missing degree squared. The label missing degree squared is added to the normalized interruption degree, and then multiplied by the connection root recurrence degree to obtain the connection root continuity potential.

[0256] For any rotating traction semantic group, the number of pixels using stable nozzle labels is counted. This number is then divided by the total number of pixels in the semantic group to obtain the percentage of stable nozzle label pixels. The normalized wheel rotation deviation of each pixel within the semantic group is negatively converted and exponentially transformed. All transformation results are summed and divided by the number of pixels in the semantic group to obtain the average wheel rotation compliance. The percentage of stable nozzle label pixels is multiplied by itself, and then multiplied by the average wheel rotation compliance to obtain the wheel rotation repulsion potential.

[0257] The potential of a single pixel is summed for all pixels to be inferred, the potential of pixel pairs of all adjacent pixels is summed, the potential of the connection roots of all root continuation semantic groups is summed, and the potential of the wheel rotation repulsion of all rotation traction semantic groups is summed. The four summation results are then multiplied by their corresponding weights and added together to obtain the total energy of the higher-order conditional random field.

[0258] The weights of each potential are set based on the nozzle positioning error and continuous frame positioning jitter in the verification image. Specifically, the weights are determined as follows: the medians of the four potentials in the verification image are statistically analyzed, and the reciprocals of each median are normalized to form the initial weights. Within a range of 0.5 to 2 times the initial weights, an adjustment step of 0.1 times the initial weights is used to iterate through the weight combinations. This adjustment step allows each weight to generate at least 10 candidate values. The average distance between the nozzle structure center and the labeled center under each weight combination, as well as the standard deviation of the nozzle structure center position after continuous frame compensation, are calculated. These two values ​​are normalized and then added together. The weight combination with the smallest result is selected.

[0259] For example, a certain continuation semantic group includes 20 pixels to be inferred, of which 16 pixels use stable nozzle labels. Dividing 16 by 20 gives the percentage of pixels with stable nozzle labels (0.80). Subtracting 0.80 from 1 gives the label missing degree (0.20). Multiplying 0.20 by itself gives 0.04. One label interruption occurs along the continuation path; dividing 1 by 20 gives the normalized interruption degree (0.05). Adding 0.04 and 0.05 gives 0.09, which is then multiplied by the connection root recurrence degree (0.86) to obtain the connection root continuation potential (0.0774), approximately 0.077. This calculation result is correct.

[0260] Step S304: Perform joint semantic inference and structure aggregation on the candidate pixels of the nozzle based on the high-order conditional random field to obtain the stable structure semantic mask sequence and the common motion attachment residual sequence of the nozzle.

[0261] In this embodiment, the single-pixel potential of each pixel to be inferred is calculated when using a stable nozzle, co-motion attachment, wheel rotation traction, and background label, and the label with the smallest single-pixel potential is taken as the initial label.

[0262] The pixels to be inferred are traversed according to the image acquisition time sequence and pixel position. For the current pixel to be inferred, its label is replaced with one of the four candidate labels in turn. The single-pixel potential, pixel pair potential, root connection potential, and wheel rotation repulsion potential associated with the current pixel are recalculated. The four types of potentials are combined according to the weights determined in step S3035 to obtain the local energy corresponding to each candidate label. The label with the minimum local energy is determined as the updated label of the current pixel.

[0263] After completing one round of traversal, count the number of pixels whose labels have changed, and divide this number by the total number of pixels to be inferred to obtain the label change ratio. Iterate and update all pixels to be inferred until the label change ratio is lower than the iteration stopping ratio, or the maximum number of iterations is reached, at which point the iteration stops.

[0264] The iteration stopping ratio and maximum number of iterations are set based on the total energy decrease process in the verification images. The specific setting method is as follows: record the total energy after each iteration, subtract the total energy of the current iteration from the total energy of the previous iteration, and then divide by the initial total energy to obtain the relative energy decrease. When the relative energy decrease continuously remains within the minimum distinguishable range allowed by the calculation accuracy, record the corresponding label change ratio and the number of iterations. The median of the label change ratios for each verification image is taken as the iteration stopping ratio, and the maximum number of iterations required to reach this state is taken as the maximum number of iterations.

[0265] Pixels ultimately labeled as "stable nozzles" are grouped according to the image acquisition time sequence to obtain a semantic mask sequence for the nozzle stability structure. Pixels ultimately labeled as "co-motion attachment" are grouped to obtain a co-motion attachment residual sequence.

[0266] For example, the maximum number of iterations is 20, and the iteration stopping ratio is 0.001. After the 11th iteration, 2 pixels out of 2500 pixels to be inferred have changed labels. Dividing 2 by 2500 gives a label change ratio of 0.0008. Since this label change ratio is less than 0.001, the iteration stops. This calculation result is correct.

[0267] Step S4: Based on the common motion attachment residual sequence, the oil nozzle stable structure semantic mask sequence, the current semantic probability map sequence, and the instance mask sequence, determine the oil nozzle structure positioning center and obtain the trolley wheel oil nozzle positioning result.

[0268] The specific steps of step S4 are as follows:

[0269] Step S401: Match the position of the co-motion attachment residual sequence with the nozzle stable structure semantic mask sequence, and remove the corresponding co-motion attachment region to obtain the nozzle structure mask.

[0270] In this embodiment, the co-motion attachment residual mask is positionally matched with the corresponding nozzle stabilization structure semantic mask according to the image acquisition sequence. The co-motion attachment residual mask is then morphologically dilated to obtain the attachment influence region.

[0271] The morphological expansion width is set based on the lateral width of the semantic mask for the nozzle stabilization structure. Specifically, the method is as follows: Based on the principal axis direction of the semantic mask for the nozzle stabilization structure, the lateral width of the mask is statistically analyzed at multiple positions perpendicular to the principal axis direction, and the median of the lateral width is calculated. For the verification image, the maximum width of the attachment residue intruding into the edge of the nozzle body is statistically analyzed, and this maximum width is divided by the corresponding lateral width of the nozzle body to obtain the attachment intrusion ratio. The maximum value of the attachment intrusion ratio in all valid verification images is taken as the expansion ratio. The morphological expansion width is obtained by multiplying the median lateral width by the expansion ratio and rounding up.

[0272] Pixels overlapping with the attachment influence region are removed from the semantic mask of the nozzle stable structure to obtain the initial nozzle structure region. Connectivity region aggregation is performed on the initial nozzle structure region, and the connected region with the largest area that connects to the nozzle shaft end is determined as the nozzle structure mask.

[0273] Step S402: Extract the shaft end instance region from the instance mask sequence, and determine the adjacent boundary between the nozzle structure mask and the shaft end instance region to obtain the nozzle shaft end connection boundary.

[0274] In this embodiment, based on the image acquisition timing corresponding to the nozzle structure mask, the shaft-end instance mask is read from the instance mask sequence to obtain the shaft-end instance region. The nozzle structure mask boundary and the shaft-end instance region boundary are then extracted respectively.

[0275] For any nozzle structure boundary pixel, calculate the horizontal and vertical differences between it and each axis end boundary pixel, add the squares of the two differences and perform a square root operation, and select the minimum distance as the axis end boundary distance of the pixel.

[0276] The connection boundary distance limit is set based on the equivalent diameter of the shaft end instance region. Specifically, the equivalent diameter of the shaft end is determined according to the equivalent diameter calculation process in step S30241. For the verification image, the distance from the farthest pixel to the shaft end boundary in the actual nozzle shaft end connection boundary is calculated, and then divided by the shaft end equivalent diameter to obtain the connection boundary distance ratio. The maximum value of the connection boundary distance ratio in all valid verification images is taken as the adoption ratio. The connection boundary distance limit is obtained by multiplying the shaft end equivalent diameter by the connection boundary distance ratio.

[0277] The nozzle structure boundary pixels whose distance from the shaft end boundary is not greater than the connection boundary distance limit are grouped into connected regions to obtain the nozzle shaft end connection boundary.

[0278] Step S403: Extract the contour of the nozzle structure mask and filter the nozzle contour by combining it with the current semantic probability map sequence to obtain the nozzle axial contour.

[0279] In this embodiment, boundary tracking is performed on the nozzle structure mask, and continuous boundary pixels are arranged in adjacency order to obtain candidate nozzle contours. The nozzle semantic probabilities of the pixels corresponding to the candidate nozzle contours are read from the current semantic probability map sequence.

[0280] The semantic probabilities of each pixel in the candidate nozzle contour are summed and then divided by the number of pixels in the contour to obtain the average nozzle semantic probability. The number of pixels in the candidate nozzle contour is counted to obtain the contour length. The number of pixels in the candidate nozzle contour that coincide with or are directly adjacent to the nozzle shaft end connection boundary is counted to obtain the number of connected pixels.

[0281] The normalized contour length and the normalized number of connected pixels are obtained by dividing the candidate contour length by the maximum contour length among all candidate contours and by dividing the number of connected pixels by the maximum number of connected pixels among all candidate contours.

[0282] The average nozzle semantic probability, normalized contour length, and normalized connected pixel count are multiplied by their respective weights and summed to obtain the contour filtering degree. Each weight is set based on the selection ratio of the real nozzle contours in the verification images. Specifically, the weights are adjusted under the condition that the sum of the three weights is 1. The number of verification images in which the real nozzle contours achieve the highest filtering degree is counted, and then divided by the total number of verification images to obtain the real contour selection ratio. The weight combination with the highest selection ratio is selected. The candidate nozzle contour with the highest contour filtering degree and connected to the nozzle axial end boundary is determined as the nozzle axial contour.

[0283] Step S404: Based on the connection boundary of the nozzle shaft end and the axial contour of the nozzle, calculate the center of the nozzle structure to obtain the positioning result of the nozzle on the trolley wheel.

[0284] In this embodiment, the abscissas of all boundary pixels in the connecting boundary of the nozzle shaft end are summed and then divided by the number of boundary pixels to obtain the abscissa of the connecting boundary center. The ordinates are processed in the same way to obtain the ordinate of the connecting boundary center.

[0285] The nozzle structure mask is peeled off layer by layer, removing outer layer pixels while maintaining regional connectivity, until a nozzle structure skeleton with a single pixel width is obtained. The distance between each skeleton point and the center of the connecting boundary is calculated, and the skeleton point with the smallest distance is taken as the skeleton starting point. Paths to each skeleton endpoint are searched along the nozzle structure skeleton, and the number of skeleton pixels contained in each path is counted. The path with the largest number of pixels is determined as the nozzle axial skeleton.

[0286] Calculate the average values ​​of the x-coordinate and y-coordinate of each frame point in the axial frame of the nozzle. Calculate the difference in lateral and longitudinal coordinates of each frame point relative to the average coordinate values. Construct a coordinate covariance matrix based on the average of the squared differences in lateral and longitudinal coordinates, and the average of their product. Perform eigenvalue decomposition on the coordinate covariance matrix, and determine the direction corresponding to the largest eigenvalue as the axial direction of the nozzle.

[0287] Calculate the distances between the two ends of the nozzle axial skeleton and the center of the connecting boundary, and determine the end with the larger distance as the outer end direction of the nozzle. For any contour pixel in the nozzle axial contour, multiply the difference in the lateral coordinates of the pixel relative to the center of the connecting boundary by the lateral component of the nozzle axial direction, multiply the difference in the longitudinal coordinates by the longitudinal component of the nozzle axial direction, and then add the two products to obtain the axial projection position.

[0288] The quantile limit for the outer contour region of the nozzle is set based on the positioning error in the verification image. The specific setting method is as follows: multiple candidate quantile values ​​are applied to the axial projection position respectively, contour pixels with a value not lower than the corresponding quantile value are extracted, the distance between the semantic probability weighting center of the extracted pixels and the center of the labeled nozzle structure is calculated, the average distance is calculated on the verification image, and the quantile value with the smallest average distance is selected.

[0289] Read the semantic probability of each pixel in the outer contour region of the nozzle. Multiply the x-coordinate of each pixel by the corresponding semantic probability of the nozzle and sum them, then divide by the total semantic probability of the nozzle to obtain the x-coordinate of the nozzle structure center. Multiply the y-coordinate of each pixel by the corresponding semantic probability of the nozzle and sum them, then divide by the total semantic probability of the nozzle to obtain the y-coordinate of the nozzle structure center.

[0290] By associating the nozzle structure center, nozzle axial direction, trolley markings, and nozzle structure mask, the nozzle positioning result of the trolley wheel is obtained.

[0291] Example 2

[0292] See Figure 3 This embodiment provides a trolley wheel nozzle positioning system based on lightweight instance segmentation, including: an element acquisition module, a same-wheel matching module, a semantic graph construction module, and a nozzle positioning module.

[0293] The element acquisition module is used to acquire the positioning element set of the trolley wheel oil nozzle.

[0294] The same-wheel matching module performs same-wheel image matching and instance segmentation based on the positioning element set of the trolley wheel oil nozzle, and obtains the current semantic probability map sequence, the historical semantic probability map sequence and the instance mask sequence.

[0295] The semantic graph construction module constructs a heterogeneous motion semantic graph of the wheel axle based on the trolley wheel running image, the current semantic probability graph sequence, the historical semantic probability graph sequence, and the instance mask sequence, and obtains the oil nozzle stable structure semantic mask sequence and the common motion attachment residual sequence.

[0296] The nozzle positioning module determines the nozzle structure positioning center based on the common motion attachment residual sequence, the nozzle stable structure semantic mask sequence, the current semantic probability map sequence, and the instance mask sequence, thus obtaining the trolley wheel nozzle positioning result.

[0297] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 1, and will not be repeated here.

[0298] In addition, the parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of the corresponding technical solutions in the prior art have not been described in detail, so as to avoid excessive elaboration.

[0299] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for positioning grease nipples on trolley wheels based on lightweight instance segmentation, characterized in that, include: Obtain the positioning element set of the trolley wheel oil nozzle, which includes historical image data, lightweight instance segmentation network, trolley operation identification information and trolley wheel operation images; Based on the positioning element set of the trolley wheel oil nozzle, image matching and instance segmentation of the same wheel are performed to obtain the current semantic probability map sequence, the historical semantic probability map sequence and the instance mask sequence. Based on the trolley wheel running image, the current semantic probability map sequence, the historical semantic probability map sequence, and the instance mask sequence, a wheel-axle heterogeneous motion semantic map is constructed to obtain the nozzle stable structure semantic mask sequence and the common motion attachment residual sequence. The nozzle stable structure semantic mask sequence and the common motion attachment residual sequence are obtained by performing motion constraints and joint inference of the nozzle candidate pixels based on the wheel-axle heterogeneous motion semantic map. Based on the common motion attachment residual sequence, the semantic mask sequence of the nozzle stable structure, the current semantic probability map sequence, and the instance mask sequence, the nozzle structure positioning center is determined, and the nozzle positioning result of the trolley wheel is obtained.

2. The method for positioning trolley wheel oil nozzles based on lightweight instance segmentation according to claim 1, characterized in that, The method involves performing same-wheel image matching and instance segmentation based on the positioning feature set of the trolley wheel nozzles, resulting in a current semantic probability map sequence, a historical semantic probability map sequence, and an instance mask sequence, including: Based on the trolley operation identification information and the trolley wheel operation images, a trolley identification is generated, and the trolley wheel operation images are arranged to obtain the current image sequence of the trolley wheels. Based on the historical image data and trolley identification, historical images of the same trolley wheels are extracted and image registration is performed to obtain a historical passing image sequence; The current image sequence and the historical passing image sequence of the trolley wheels are input into the lightweight instance segmentation network for multi-scale feature extraction and fusion to obtain the current semantic probability map sequence, the historical semantic probability map sequence, the instance center response and the center offset. The foreground pixels in the current semantic probability map sequence and the historical semantic probability map sequence are determined, and instances are grouped according to the instance center response and center offset to obtain the instance mask sequence.

3. The method for positioning trolley wheel oil nozzles based on lightweight instance segmentation according to claim 1, characterized in that, The construction of the heterogeneous motion semantic graph of the wheel axle yields a semantic mask sequence of the nozzle stable structure and a common motion attachment residual sequence, including: Based on the trolley wheel running image, the current semantic probability map sequence and the instance mask sequence, the common translation of the axle end is extracted and the wheel body rotation phase is disambiguated to obtain the axle end translation flow field and the wheel body phase disambiguated rotation flow field. Based on the trolley operation identification information, the current semantic probability map sequence, the historical semantic probability map sequence, the instance mask sequence, and the flow field, a wheel-axle heterogeneous motion semantic map is constructed. Based on the heterogeneous motion semantic map of the wheel and axle and the flow field, candidate pixels of the nozzle are determined, and root continuation semantic clique and rotational traction semantic clique are constructed to obtain a high-order conditional random field. Based on high-order conditional random fields, joint semantic inference and structural aggregation are performed on candidate pixels of the nozzle to obtain a stable structural semantic mask sequence and a common motion attachment residual sequence of the nozzle.

4. The method for positioning trolley wheel oil nozzles based on lightweight instance segmentation according to claim 3, characterized in that, The extraction of the shaft end through common translation and wheel rotation phase disambiguation includes: Bidirectional dense optical flow estimation is performed on the images of adjacent trolley wheels to obtain forward dense optical flow, reverse dense optical flow and bidirectional pixel correspondence confidence; Extract the axle end instance region and wheel body instance region, and combine the axle end semantic probability and bidirectional pixel correspondence confidence to perform weighted aggregation and residual correction on the forward dense optical flow to obtain the axle end common translation amount and axle end translation flow field. Extract the semantic boundary response and image gradient response of the wheel body within the wheel body instance region, and perform circumferential expansion and radial aggregation with the center of the axle end instance as the reference to obtain the circumferential semantic expansion sequence; The circumferential semantic unfolding sequence of the next frame is corrected based on the common translation of the axle end, and cyclic correlation matching is performed with the circumferential semantic unfolding sequence of the previous frame to obtain the wheel rotation phase candidate and the degree of cyclic correlation. By combining the forward dense optical flow, the reverse dense optical flow, and the common translation at the shaft end, the candidate rotating phase of the wheel body is verified by rotation and screened for continuity, resulting in the wheel body phase disambiguation rotating flow field.

5. The method for positioning trolley wheel oil nozzles based on lightweight instance segmentation according to claim 4, characterized in that, The rotation verification and continuity screening of the candidate wheel rotation phases includes: Based on the rotation phase candidates of each wheel body, the pixels in the wheel body instance area are rotated around the center of the shaft end instance and the common translation of the shaft end is superimposed to obtain the wheel body candidate motion flow field; By comparing the candidate motion flow field of the wheel body with the forward dense optical flow, the candidate motion residuals are obtained, and the reverse dense optical flow is used for reverse mapping to obtain the difference between forward and reverse rotational closure. Based on the candidate motion residuals, the difference in forward and reverse rotation closure, and the degree of cyclic correlation, the first group of target rotation phases is selected. Based on the image order and the continuity difference between the current candidate and the previous group of target rotation phases, the remaining target rotation phases are filtered. Based on the rotational phase of each group of targets and the common translation of the shaft end, the rotational translation mapping of the wheel body instance region is performed to obtain the wheel body phase-disambiguated rotating flow field.

6. The method for positioning trolley wheel oil nozzles based on lightweight instance segmentation according to claim 3, characterized in that, The construction of the heterogeneous motion semantic graph of wheel and axle includes: Extract historical and current shaft end instances, wheel body instances, and oil nozzle candidate instances from the current semantic probability graph sequence, historical semantic probability graph sequence, and instance mask sequence, establish corresponding nodes, and obtain a heterogeneous semantic node set; By mapping the historical shaft end instance using the shaft end translation flow field, and based on the degree of overlap and positional deviation between the mapped region and the current shaft end instance, a shaft end translation receiving edge is established; By using the phase-disambiguated rotating flow field mapping of wheel body instances to map historical wheel body instances, and based on the degree of overlap and semantic distribution differences between the mapped region and the current wheel body instance, a wheel body rotation bearing edge is established. Based on the trolley operation identification information, extract the historical connection root region and the current connection root region of the same trolley wheel, and establish the connection root reproduction edge through axle end benchmark alignment and reproduction comparison; By organizing the heterogeneous semantic node set and various edges into a graph structure, a heterogeneous motion semantic graph of wheels and axles is obtained.

7. The method for positioning trolley wheel oil nozzles based on lightweight instance segmentation according to claim 6, characterized in that, The alignment and replication comparison of the shaft end reference, and the establishment of the connection root replication edge, include: Based on the connection position between the nozzle candidate instance mask and the shaft end instance mask, extract the current connection root region; Based on the trolley operation identification information, extract the historical oil nozzle candidate instance mask and historical axle end instance mask of the same trolley wheel from the historical semantic probability map sequence and instance mask sequence to determine the historical connection root region; Determine the current axis-end geometric reference and the historical axis-end geometric reference based on the axis-end center, region size, and main axis direction of the current axis-end instance mask and the historical axis-end instance mask, respectively; Construct the current axis-end local coordinate system based on the geometric datum, and map the historical connection root region to the local coordinate system to obtain the aligned historical connection root region; Compare the current connection root region with the aligned historical connection root region to determine the degree of connection root repetition and establish connection root repetition edges.

8. The method for positioning trolley wheel oil nozzles based on lightweight instance segmentation according to claim 3, characterized in that, The process involves determining nozzle candidate pixels based on the heterogeneous motion semantic map of the wheel and axle and the flow field, constructing a root continuation semantic clique and a rotational traction semantic clique, and obtaining a high-order conditional random field, including: Candidate pixels for the nozzle are determined from the current semantic probability map sequence and instance mask sequence. Semantic probabilities of the nozzle, shaft end and wheel body are extracted to obtain the set of pixels to be inferred. Motion mapping of the pixel to be inferred is performed using the translational flow field at the shaft end and the phase-disambiguated rotating flow field of the wheel body to construct the single-pixel potential and the pixel-pair potential. The association between the pixel to be inferred and the axle end instance is determined based on the heterogeneous motion semantic graph of the wheel and axle, and the axle end structure continuation pixels are collected to obtain the root continuation semantic group. The association between the pixel to be inferred and the wheel instance is determined based on the heterogeneous motion semantic map of the wheel axle and the phase disambiguation rotating flow field of the wheel body. The rotating mapping continuous pixels are collected to obtain the rotating traction semantic group. Based on two types of semantic groups, a connection root continuation potential and a wheel rotation repulsion potential are constructed, and combined with a single pixel potential and a pixel pair potential to obtain a higher-order conditional random field.

9. The method for positioning trolley wheel oil nozzles based on lightweight instance segmentation according to claim 1, characterized in that, The method of determining the nozzle structure positioning center based on the common motion attachment residual sequence, the nozzle stable structure semantic mask sequence, the current semantic probability map sequence, and the instance mask sequence, and obtaining the trolley wheel nozzle positioning result, includes: The positions of the co-motion attachment residual sequence and the nozzle stable structure semantic mask sequence are matched, and the corresponding co-motion attachment regions are removed to obtain the nozzle structure mask. Extract the shaft end instance region from the instance mask sequence and determine the adjacent boundary between the nozzle structure mask and the shaft end instance region to obtain the nozzle shaft end connection boundary; The contour of the nozzle structure mask is extracted and filtered by combining the current semantic probability map sequence to obtain the axial contour of the nozzle. Based on the connection boundary of the nozzle shaft end and the axial contour of the nozzle, the center of the nozzle structure is calculated to obtain the positioning result of the nozzle on the trolley wheel.

10. A trolley wheel grease nipple positioning system based on lightweight instance segmentation, used to implement the trolley wheel grease nipple positioning method based on lightweight instance segmentation as described in any one of claims 1 to 9, characterized in that, include: The element acquisition module is used to acquire the positioning element set of the trolley wheel oil nozzles; The same-wheel matching module performs same-wheel image matching and instance segmentation based on the positioning element set of the trolley wheel oil nozzle, and obtains the current semantic probability map sequence, the historical semantic probability map sequence and the instance mask sequence; The semantic graph construction module constructs a semantic graph of heterogeneous motion of wheel and axle based on the trolley wheel running image, the current semantic probability graph sequence, the historical semantic probability graph sequence and the instance mask sequence, and obtains the semantic mask sequence of the oil nozzle stable structure and the common motion attachment residual sequence. The nozzle positioning module determines the nozzle structure positioning center based on the common motion attachment residual sequence, the nozzle stable structure semantic mask sequence, the current semantic probability map sequence, and the instance mask sequence, thus obtaining the nozzle positioning result of the trolley wheel.

Images