A computer vision-based method for inspecting highway maintenance construction equipment

Abstract

The application discloses a kind of based on computer vision's highway maintenance construction equipment's inspection method, it is related to highway maintenance construction equipment intelligent detection technical field, including, acquisition image frame data, equipment event data and control command data, carry out time synchronization processing and space coordinate calibration, identify equipment component image area in image frame data, generate equipment component image segment, extract structure point coordinates to equipment component image segment, calculate three-dimensional pose parameter, generate equipment component three-dimensional pose sequence and construct action state vector, equipment event data and control command data are converted into control command semantic vector, based on control command semantic vector and action state vector, calculate the action behavior score value of equipment component image segment, for the equipment component image segment of action behavior score value below action behavior score threshold, structure state evaluation is carried out, generates structured detection report.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection technology for highway maintenance and construction equipment, and in particular to an inspection method for highway maintenance and construction equipment based on computer vision. Background Technology

[0002] In the field of intelligent inspection technology for highway maintenance and construction equipment, image recognition and computer vision technologies have been widely applied in industrial equipment inspection, enabling non-contact, real-time target state recognition and motion behavior monitoring. Computer vision algorithms, represented by deep learning, have achieved breakthroughs in target detection, pose estimation, and semantic segmentation, and are gradually being applied to real-time fault diagnosis and behavior monitoring of industrial equipment, enabling the perception and early warning of on-site equipment operating status within milliseconds.

[0003] However, existing methods still have two limitations: most schemes focus on single-frame defect detection of static images or point clouds, and fail to achieve three-dimensional attitude tracking of component displacement and rotation changes in continuous frames. They also lack subsequent structural state assessment for detected motion anomalies and cannot provide a structured detection report that includes component motion state and risk level. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a computer vision-based inspection method for highway maintenance and construction equipment, which solves the problems of existing technologies being unable to achieve continuous frame three-dimensional posture tracking and lacking subsequent structural state assessment and structured inspection report generation for equipment with abnormal movements.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for inspecting highway maintenance and construction equipment based on computer vision, comprising,

[0008] Acquire image frame data, device event data, and control command data; perform time synchronization processing and spatial coordinate calibration on the image frame data, device event data, and control command data.

[0009] Identify device component image regions in image frame data, extract device component image regions, and generate device component image fragments;

[0010] Extract the coordinates of structural points from image fragments of equipment components, arrange them in chronological order to form a three-dimensional trajectory sequence of structural points, calculate the three-dimensional attitude parameters of the equipment components, generate a three-dimensional attitude sequence of the equipment components, and construct an action state vector.

[0011] The control command data and device event data are converted into control command semantic vectors. Based on the control command semantic vectors and action state vectors, the action behavior score of the device component image fragment is calculated, and it is determined whether the action behavior score is lower than the action behavior score threshold.

[0012] For image fragments of equipment components whose action behavior scores are lower than the action behavior score threshold, a structural health score is calculated to assess the structural status. The risk level of the equipment component is calculated by combining the action behavior score and the structural health score, and a structured inspection report is generated.

[0013] As a preferred embodiment of the computer vision-based inspection method for highway maintenance and construction equipment described in this invention, the specific steps for acquiring image frame data, equipment event data, and control command data are as follows:

[0014] The system collects image frame data, equipment event data, and control command data during the operation of highway maintenance and construction equipment. The image frame data is acquired by an industrial camera, while the equipment event data and control command data are acquired by the vehicle bus interface.

[0015] As a preferred embodiment of the computer vision-based inspection method for highway maintenance and construction equipment described in this invention, the specific steps for performing time synchronization processing and spatial coordinate calibration on image frame data, equipment event data, and control command data are as follows:

[0016] Time synchronization processing is performed on image frame data, device event data, and control command data. The comprehensive time deviation is calculated, and the time synchronization threshold is obtained based on the frame rate interval of the industrial camera and the peak value of the vehicle bus timestamp jitter. The comprehensive time deviation is compared with the time synchronization threshold to determine whether the synchronization is successful.

[0017] The coordinates of calibration points are extracted from the images of equipment components, and three-dimensional spatial points in the industrial camera coordinate system are obtained by combining them with the actual calibration plate size. The coordinates of the calibration points are then mapped to three-dimensional world coordinates to complete the spatial coordinate calibration.

[0018] As a preferred embodiment of the computer vision-based inspection method for highway maintenance and construction equipment described in this invention, the specific steps of identifying equipment component image regions in image frame data, extracting equipment component image regions, and generating equipment component image fragments are as follows:

[0019] Device component identification is performed on image frame data to generate candidate box pixel coordinates, candidate box width and candidate box height, the confidence score of the candidate box is calculated, redundant candidate boxes are filtered out based on the confidence score, and the filtered candidate boxes are obtained.

[0020] By filtering candidate box pixel coordinates, candidate box height, and candidate box width, the device component image region is extracted from the device component image, and a device component image fragment is generated.

[0021] As a preferred embodiment of the computer vision-based inspection method for highway maintenance and construction equipment described in this invention, the specific steps of extracting structural point coordinates from image fragments of equipment components, arranging them in chronological order to form a three-dimensional trajectory sequence of structural points are as follows:

[0022] Two-dimensional structural points are extracted from the image fragments of each device component by structural point detection, and the coordinates of the structural points are obtained by three-dimensional back projection of the two-dimensional structural points.

[0023] The coordinates of the structural points extracted from each frame are arranged in chronological order to form a three-dimensional trajectory sequence of the structural points.

[0024] As a preferred embodiment of the computer vision-based inspection method for highway maintenance and construction equipment described in this invention, the specific steps for calculating the three-dimensional attitude parameters of the equipment components, generating a three-dimensional attitude sequence of the equipment components, and constructing an action state vector are as follows:

[0025] Based on the three-dimensional trajectory sequence of structural points, the spatial attitude of the device component in consecutive frames is calculated by fitting the spatial rigid centroid with the attitude axis, and the three-dimensional attitude sequence of the device component is generated.

[0026] By using frequency-band Fourier transform, the three-dimensional attitude sequence of the equipment components is encoded to generate an action state vector.

[0027] As a preferred embodiment of the computer vision-based inspection method for highway maintenance and construction equipment described in this invention, the steps include: converting control command data and equipment event data into control command semantic vectors, and calculating the action behavior score of equipment component image fragments based on the control command semantic vectors and action state vectors.

[0028] The control command data and device event data are combined and transformed into a control command semantic vector through a semantic encoding function;

[0029] Calculate the normalized cosine similarity between the action state vector and the control command semantic vector to obtain the action behavior score of the device component image fragment.

[0030] In a preferred embodiment of the computer vision-based inspection method for highway maintenance and construction equipment described in this invention, the specific steps for determining whether the action behavior score is lower than the action behavior score threshold are as follows:

[0031] Select normal samples that are confirmed to be without abnormalities under standard operating conditions and abnormal samples that are confirmed to be under typical fault operating conditions.

[0032] Calculate the normal action behavior score of normal equipment and the abnormal action behavior score of abnormal equipment, and construct a probability density function;

[0033] The threshold for action behavior scoring is obtained from the cross-distribution interval of the probability density function of normal action behavior score and the probability density function of abnormal action behavior score;

[0034] Determine whether the action behavior score is lower than the action behavior score threshold.

[0035] As a preferred embodiment of the computer vision-based inspection method for highway maintenance and construction equipment described in this invention, the specific steps for calculating structural health scores and assessing structural condition for image fragments of equipment components with action behavior scores below a certain threshold are as follows:

[0036] Texture analysis is performed on image fragments of device components whose action behavior scores are lower than the action behavior score threshold to obtain texture feature vectors;

[0037] A structural response map is generated based on the structural point coordinates of image fragments of equipment components. The structural health score is calculated by combining the texture feature vector with the structural response map to assess the structural status.

[0038] As a preferred embodiment of the computer vision-based inspection method for highway maintenance and construction equipment described in this invention, the steps for calculating the risk level of equipment components by combining action behavior scores and structural health scores to generate a structured inspection report are as follows:

[0039] Risk assessment indicators are constructed based on behavioral and structural health scores, and risk levels are determined through quantile analysis.

[0040] The behavioral scores, structural health scores, and risk levels are summarized and organized to generate a structured assessment report.

[0041] The beneficial effects of this invention are as follows: Based on the three-dimensional trajectory sequence of structural points, a three-dimensional posture sequence and action state vector of equipment components are generated, realizing high-precision continuous tracking of dynamic posture changes of construction equipment components; control command data and equipment event data are converted into control command semantic vectors and matched with action state vectors to calculate action behavior score values; and a structured detection report is generated for image segments with action behavior score values ​​lower than the action behavior score threshold, realizing real-time detection of abnormal actions. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a computer vision-based inspection method for highway maintenance and construction equipment.

[0044] Figure 2 This is a flowchart for multi-source data acquisition and synchronous calibration.

[0045] Figure 3 This is a flowchart for the three-dimensional attitude analysis of equipment components.

[0046] Figure 4 A flowchart for risk assessment and reporting. Detailed Implementation

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0049] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0050] Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides a computer vision-based inspection method for highway maintenance and construction equipment, comprising the following steps:

[0051] S1. Acquire image frame data, device event data, and control command data; perform time synchronization processing and spatial coordinate calibration on the image frame data, device event data, and control command data.

[0052] The status information of highway maintenance and construction equipment during operation is collected through multi-sensor collaboration. The collected status information includes: image frame data, equipment event data, and control command data.

[0053] Image frame data is acquired by industrial cameras installed on highway maintenance and construction equipment, while equipment event data and control command data are collected in real time by the vehicle bus interface and are automatically marked with timestamps.

[0054] Time synchronization processing is performed on image frame data, device event data, and control command data. A weighted linear time alignment strategy is adopted, using the timestamp of image frame data as a reference, and weighted alignment is performed on the timestamps of control command data and device event data to obtain the comprehensive time deviation.

[0055] Choosing a 30fps industrial camera is sufficient to accurately capture the displacement and rotation changes of components in highway maintenance equipment. At the same time, the generated data can be processed in real time by edge computing nodes. The peak jitter of the vehicle bus timestamp is about 10ms. The synchronization tolerance is obtained based on the peak jitter of the vehicle bus timestamp and the two-frame interval corresponding to the frame rate. If the synchronization tolerance is too large, it will affect the real-time performance. If the synchronization tolerance is too small, a large number of valid frames may be discarded due to small jitter. Multiple control command timestamps and device event record timestamps are collected. The standard deviation of the control command timestamp and the standard deviation of the device event record timestamp are calculated to construct the synchronization deviation tolerance and obtain the synchronization threshold.

[0056] If the overall time deviation is less than the synchronization threshold, the data time synchronization is considered successful; if the overall time deviation is greater than the synchronization threshold, the current frame data is discarded.

[0057] After time synchronization processing is completed, spatial coordinate calibration is performed on the image frame data. Zhang's calibration method is adopted. By acquiring a checkerboard calibration board through multiple images, the coordinates of calibration points are extracted from the image frame data. Combined with the actual calibration board size, the intrinsic parameter matrix, rotation matrix and displacement vector of the industrial camera are obtained.

[0058] The unit direction vector in the industrial camera coordinate system is obtained by inverse operation of the intrinsic parameter matrix. Based on the industrial camera, a depth map is generated by a stereo vision algorithm. The unit direction vector is scaled according to the depth value of the corresponding pixel to obtain the three-dimensional spatial point in the industrial camera coordinate system. First, the displacement vector is subtracted from the three-dimensional spatial point to eliminate the offset of the camera optical center in the world coordinate system. Then, the inverse rotation of the rotation matrix is ​​applied to the three-dimensional spatial point to restore the world coordinate system orientation. The calibration point coordinates of the image frame data are mapped to three-dimensional world coordinates to complete the spatial coordinate calibration.

[0059] S2. Identify the device component image region in the image frame data, extract the device component image region, and generate device component image fragments;

[0060] The improved YOLOv7-Tiny target detection algorithm is used to identify equipment components in image frame data. Target detection is performed on the image region of highway maintenance and construction equipment components in each image frame data, the image region of the equipment component is located, and the image region of the equipment component is extracted to generate equipment component image fragments.

[0061] Specifically, a component image dataset covering key components of highway maintenance and construction equipment under different conditions is constructed for training. A dual-scale attention (DSA) mechanism is embedded in the convolutional neural network structure to form an improved YOLOv7-Tiny initial model. A data augmentation pipeline including spatial, scale, color, and mixed sampling is established and multi-scale training is enabled. Forward propagation is performed according to the improved network topology. The response of key feature layers is enhanced by DSA. A joint loss function consisting of bounding box regression, targetability, and classification branches is adopted. Imbalance and noise are suppressed by combining label smoothing and sample reweighting strategies, and the three loss terms are weighted. Backpropagation and parameter updates are performed and a stabilization strategy is maintained. The detection performance is periodically evaluated on the validation set, the index is recorded, and early stopping and optimal model selection are implemented based on convergence. Finally, the weights are solidified, the inference and post-processing parameters are configured, and the improved YOLOv7-Tiny convolutional neural network is derived.

[0062] Each image frame data is input into the improved YOLOv7-Tiny convolutional neural network to obtain the raw detection prediction of the image frame. By decoding the raw detection prediction of the image frame and mapping it to the image coordinate system, candidate boxes are output, with the expression as follows:

[0063] ;

[0064] in, Indicates the first One candidate box, This represents the pixel coordinates of the top-left corner of the candidate bounding box in the original image. Represents the x-coordinate of the candidate box. Represents the ordinate of the candidate box. Indicates the width of the candidate box. This indicates the candidate box height, used to determine the position and size of the candidate box in the image;

[0065] The candidate box index position is obtained by outputting the tensor of the detection head of the improved YOLOv7-Tiny convolutional neural network. The original object score and the original category response score of the candidate box are extracted. The original object score is input into the Sigmoid transformation function to obtain the object confidence. The larger the original object score, the closer the object confidence is to 1, and the greater the probability that the candidate box contains the target. The smaller the original object score, the closer the object confidence is to 0, and the smaller the probability that the candidate box contains the target.

[0066] Object confidence reflects whether any highway maintenance or construction equipment components exist in the image area covered by the candidate bounding box;

[0067] The original category response scores are exponentially amplified, and the amplified original category response scores are normalized using Softmax to obtain the category probability distribution. The category probability is then obtained based on the category probability distribution.

[0068] The category probability reflects the likelihood that a candidate bounding box belongs to a specific component category, assuming that the image region it covers contains a component.

[0069] The joint confidence score is obtained based on the object confidence score and the class probability. To remove redundant candidate boxes caused by overlapping candidate boxes, all candidate boxes are sorted from high to low according to the joint confidence score. The candidate box with the highest joint confidence score is selected sequentially based on the Soft-NMS algorithm. The intersection-union ratio (IUR) is calculated between the candidate box with the highest joint confidence score and each of the other candidate boxes. The joint confidence score is continuously reduced according to the IUR value using the Gaussian decay function. Through iteration, redundant candidate boxes are removed.

[0070] For each candidate box that passes the screening, read the pixel coordinates of the top left corner in the original image, as well as the width and height of the candidate box, and perform boundary verification. Cut off the part that exceeds the boundary of the candidate box in the original image to the very edge of the image to ensure that all cropped areas are completely within the image range. Extract the device component image region, and use the candidate box after boundary verification as the reference box to directly crop the corresponding rectangular area from the device component image region as the device component image fragment. Add a timestamp and device identification number, and store it in the cache queue.

[0071] S3. Extract the coordinates of structural points from the image fragments of the equipment components, arrange them in chronological order to form a three-dimensional trajectory sequence of structural points, calculate the three-dimensional attitude parameters of the equipment components based on the three-dimensional trajectory sequence of structural points, generate a three-dimensional attitude sequence of the equipment components, and construct an action state vector based on the three-dimensional attitude sequence of the equipment components.

[0072] Retrieve consecutive device component image segments from the cache queue in ascending order of timestamp. For example, retrieve consecutive 5-15 frame device component image segments from the cache queue in ascending order of timestamp. If there are fewer than 5 frames, it is difficult to capture enough motion change features. If there are more than 15 frames, too much redundant information will be introduced.

[0073] By labeling pixel coordinates on image fragments of device components, two-dimensional structural points are obtained and a real structural point set with two-dimensional pixel coordinates is formed. The real structural point set is mapped through heatmap resolution and converted into a supervised heatmap based on Gaussian response. After inputting image fragments of device components, HRNet-Mechanicalv2 generates a predicted heatmap. A two-dimensional heatmap regression method is adopted, with the mean square error between the predicted heatmap and the supervised heatmap as the loss. Combined with a learning rate decay strategy, iterative training is performed. The localization error and average pixel error are calculated on the validation set. The optimal weights are selected and fixed based on the validation results. The performance is evaluated by the localization error and average pixel error during the testing phase, resulting in the re-labeled and trained HRNet-Mechanicalv2 network.

[0074] The HRNet-Mechanicalv2 network, which has been re-annotated and trained, is called on the image fragments of device components to automatically detect and output the two-dimensional pixel coordinates of two-dimensional structural points in the image fragments of device components.

[0075] Based on the actual depth of each two-dimensional structural point from the imaging plane of the industrial camera, as well as the intrinsic and extrinsic parameters of the industrial camera, the coordinates of each two-dimensional pixel are mapped to three-dimensional spatial coordinates through a three-dimensional back projection method.

[0076] Arrange the three-dimensional spatial coordinates in chronological order to form a sequence of three-dimensional trajectories of structural points;

[0077] The three-dimensional trajectory sequence of structural points is regarded as a rigid body motion trajectory. The spatial attitude of the component is calculated frame by frame by using the rigid body registration method to obtain the three-dimensional attitude sequence of the equipment component.

[0078] Multi-band discrete Fourier transform is performed on the three-dimensional trajectory sequence of structural points and the three-dimensional attitude sequence of equipment components to extract the spectral feature coefficients of displacement and attitude. The spectral feature coefficients are then concatenated in the order of displacement first and attitude second to obtain the motion state vector.

[0079] S4. Convert control command data and device event data into control command semantic vectors. Based on the control command semantic vectors and action state vectors, calculate the action behavior score and determine whether the action behavior score is lower than the action behavior score threshold.

[0080] Each control command and device event data is decoded, and the original records are uniformly mapped to a standard field set with unified time zones and units of measurement. Based on the standard field set and the command template library, action predicates and parameter slots are determined to segment time zones. Command items are generated for each segment and converted into control command semantic vectors using a semantic encoding function. The expression is as follows:

[0081] ;

[0082] in, Indicates the first The control command semantic vector of the frame, Indicates the first The number of command items split from a frame. Indicates the first The first frame One command item, The semantic encoding function represents the control command data using one-hot encoding as a command one-hot vector, and performs linear normalization on the device event data to obtain the event feature vector. The semantic vector is then obtained by sorting the command one-hot vector and the event feature vector according to the order of their fields. This represents the normalization factor, used to balance the scale differences caused by different numbers of command items;

[0083] For each frame of the device component image sequence, the first... For each frame, the normalized cosine similarity between the action state vector and the control command semantic vector is calculated, denoted as the inter-frame matching function, and its expression is:

[0084] ;

[0085] in, Indicates the first The degree of matching between the frame control command semantic vector and the action state vector. Indicates the first The frame's action state vector;

[0086] For the selected sequence of images of equipment components, a weighted summation method is used to calculate the action behavior score, expressed as:

[0087] ;

[0088] in, This indicates the score for the action / behavior. Indicates the length of the selected time window. Indicates the first The weight coefficients of each frame are non-negative and normalized. The energy metric of the action state vector for each frame is calculated, following the rule that a higher energy metric corresponds to a higher weight. This energy metric is then normalized within the device component image sequence to obtain... The weighting coefficient, as a proportion in the weighted average, must be between 0 and 1, and its value range is... ;

[0089] Several normal samples, confirmed by equipment manufacturers and field engineers under standard operating conditions, and several abnormal samples, confirmed by typical fault conditions and monitoring alarm records, are selected. Action behavior scores are calculated, and a probability density function is obtained by fitting the action behavior scores of normal samples and abnormal samples. The probability distribution of action behavior scores is obtained, and the probability distributions of normal sample action behavior scores and abnormal sample action behavior scores are superimposed to obtain the intersection of the probability density functions, which is the boundary between the false alarm rate and the missed detection rate. The range is expanded with the intersection of the probability density functions as the center of the interval. Based on reducing the missed detection of abnormalities while minimizing false alarms for normal actions, the threshold range of action behavior scores is determined.

[0090] Determine if the action behavior score is lower than the action behavior score threshold. If the action behavior score is higher than the action behavior score threshold, the equipment component is in normal condition and no further operation is required.

[0091] S5. For image segments of equipment components with action behavior scores lower than the action behavior score threshold, calculate the structural health score, conduct a structural status assessment, and combine the action behavior score and the structural health score to calculate the risk level of the equipment component and generate a structured inspection report.

[0092] When the action behavior score is lower than the action behavior score threshold, the equipment component has abnormal action. For the identified abnormal action equipment component, the comprehensive time deviation and three-dimensional posture sequence corresponding to the equipment component image segment are called to ensure the spatiotemporal consistency of the structural evaluation. Then, texture analysis is performed on the equipment component image segment, and texture features are extracted by jointly using Local Binary Mode (LBP) and Gray-Level Co-occurrence Matrix (GLCM) to obtain texture feature vector.

[0093] Furthermore, a structural response map (SRM) is generated based on the changes in structural point coordinates. Displacement differences are calculated for the same structural point in consecutive image frames. High-pass filtering is applied to the local neighborhood differences of all structural points, and the filtered response is mapped to a pseudo-color image to form the SRM. This assists in detecting subtle damage, deformation, or fatigue signs. Combining texture feature vectors with the SRM, a weighted fusion of texture consistency and structural response consistency is used to calculate the structural health score, expressed as:

[0094] ;

[0095] in, Indicates the structural health score. The structural texture feature vector is extracted jointly by the local binary pattern and the gray-level co-occurrence matrix. The structural influence quantity is obtained by mapping the local deformation response of the SRM after high-pass filtering.

[0096] Based on behavioral scores and structural health scores The risk assessment indicators for constructing equipment components are expressed as follows:

[0097] ;

[0098] in, This represents risk assessment indicators and behavioral score. With structural health score Normalized to Based on the risk scaling function, behavioral scores and structural health scores are mapped to a single risk scale, outputting a risk assessment index. According to the complementary geometric mean principle, the output range of the risk scaling function is... Range of values ​​for risk assessment indicators ;

[0099] Map risk assessment indicators to specific levels;

[0100] when This is considered low risk.

[0101] when It is classified as medium risk;

[0102] when This is considered high-risk.

[0103] when This is considered extremely high risk.

[0104] The interval division is based on the statistical analysis of risk assessment indicators and actual failure rates in historical monitoring data and the comprehensive judgment of engineering inspection. It collects and labels risk assessment indicators and corresponding actual failure situations of multiple highway maintenance and construction equipment under various working conditions such as normal maintenance, minor failure and serious failure.

[0105] interval The lower portion of the coverage indicates that the action and structure are close to normal, representing low risk; the interval A slight deviation indicates an initial anomaly, suggesting a medium risk level; [Range] A combination of significant behavioral changes and structural anomalies indicates high risk; (Interval) This indicates a high degree of coupling between the two anomalies in action and structure, signifying an extremely high risk.

[0106] The behavioral scores, structural health scores, and risk levels are summarized and organized to generate a structured assessment report.

[0107] In summary, this invention achieves high-precision continuous tracking of dynamic attitude changes of construction equipment components by: calculating and generating three-dimensional posture sequences and action state vectors of equipment components based on three-dimensional trajectory sequences of structural points; converting control command data and equipment event data into control command semantic vectors and matching them with action state vectors to calculate action behavior scores; and generating structured detection reports for image segments with action behavior scores lower than the action behavior score threshold, thereby achieving real-time detection of abnormal actions.

[0108] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for inspection of highway maintenance construction equipment based on computer vision, characterized by: The application relates to a method for generating a structured detection report of a highway maintenance construction device. The method comprises the following steps: collecting image frame data, device event data and control command data, and performing time synchronization processing and space coordinate calibration on the image frame data, the device event data and the control command data; recognizing a device component image area in the image frame data, extracting the device component image area, and generating a device component image segment; extracting structure point coordinates of the device component image segment, performing time sequence arrangement, forming a structure point three-dimensional trajectory sequence, calculating three-dimensional posture parameters of the device component, generating a device component three-dimensional posture sequence and constructing a motion state vector; converting the control command data and the device event data into a control command semantic vector, calculating a motion behavior score value of the device component image segment based on the control command semantic vector and the motion state vector, and judging whether the motion behavior score value is lower than a motion behavior score threshold value; 2. The computer vision-based inspection method of highway maintenance construction equipment as claimed in claim 1, wherein: calculating a structure health score value of the device component image segment with the motion behavior score value lower than the motion behavior score threshold value, performing structure state evaluation, combining the motion behavior score value and the structure health score value to calculate a risk level of the device component, and generating a structured detection report. The collecting of the image frame data, the device event data and the control command data comprises the following steps:

3. The computer vision-based inspection method for highway maintenance construction equipment according to claim 2, characterized in that: collecting image frame data, device event data and control command data in the running process of a highway maintenance construction device, wherein the image frame data is obtained by an industrial camera, and the device event data and the control command data are obtained by a vehicle bus interface. The time synchronization processing and space coordinate calibration on the image frame data, the device event data and the control command data comprises the following steps: performing time synchronization processing on the image frame data, the device event data and the control command data, calculating a comprehensive time deviation, obtaining a time synchronization threshold value based on an industrial camera frame rate interval and a vehicle bus time stamp jitter peak value, comparing the comprehensive time deviation with the time synchronization threshold value, and judging whether the synchronization is successful; 4. The computer vision-based inspection method for highway maintenance construction equipment of claim 3, wherein: extracting calibration point coordinates in the device component image, obtaining a three-dimensional space point in an industrial camera coordinate system in combination with an actual calibration board size, mapping the calibration point coordinates into three-dimensional world coordinates, and completing the space coordinate calibration. The recognizing of the device component image area in the image frame data, the extracting of the device component image area and the generating of the device component image segment comprise the following steps: performing device component recognition on the image frame data, generating candidate box pixel coordinates, candidate box width and candidate box height, calculating the confidence of the candidate box, screening out redundant candidate boxes based on the confidence, and obtaining screened candidate boxes; 5. The computer vision-based inspection method for highway maintenance construction equipment as claimed in claim 4, wherein: extracting the device component image area in the device component image through the screened candidate box pixel coordinates, the screened candidate box height and the screened candidate box width, and generating the device component image segment. The extracting of the structure point coordinates of the device component image segment and the time sequence arrangement comprise the following steps: extracting two-dimensional structure points of each device component image segment through structure point detection, and performing three-dimensional back projection on the two-dimensional structure points to obtain structure point coordinates; performing time sequence arrangement on the extracted structure point coordinates of each frame to form a structure point three-dimensional trajectory sequence.

6. The computer vision-based inspection method for highway maintenance construction equipment as claimed in claim 5, wherein: The three-dimensional pose parameters of the computing device component, generating a device component three-dimensional pose sequence and constructing a motion state vector, the specific steps are, Based on the three-dimensional trajectory sequence of the structure points, the spatial pose of the device component in the continuous frames is calculated by fitting the spatial rigid body center of gravity and the pose axis, and a device component three-dimensional pose sequence is generated; The device component three-dimensional pose sequence is encoded by frequency division Fourier transform to generate a motion state vector.

7. The computer vision-based inspection method for highway maintenance construction equipment as claimed in claim 6, wherein: The control command data and device event data are converted into a control command semantic vector based on the control command semantic vector and the motion state vector, and the motion behavior score value of the device component image segment is calculated, the specific steps are, The control command data and device event data are combined and converted into a control command semantic vector through a semantic encoding function; The normalized cosine similarity between the motion state vector and the control command semantic vector is calculated to obtain the motion behavior score value of the device component image segment.

8. The computer vision-based inspection method of highway maintenance construction equipment as claimed in claim 7, wherein: The specific steps of judging whether the motion behavior score value is lower than the motion behavior score threshold are, Select normal samples confirmed to be normal under standard working conditions and abnormal samples confirmed under typical fault working conditions; Calculate the normal motion behavior score value of the normal device and the abnormal motion behavior score value of the abnormal device to form a probability density function; The motion behavior score threshold is obtained from the intersection distribution interval of the normal motion behavior score value probability density function and the abnormal motion behavior score value probability density function; Judge whether the motion behavior score value is lower than the motion behavior score threshold.

9. The computer vision-based inspection method for highway maintenance construction equipment as claimed in claim 8, wherein: The specific steps of calculating the structure health score value of the device component image segment with the motion behavior score value lower than the motion behavior score threshold and performing structure state evaluation are, Texture analysis is performed on the device component image segment with the motion behavior score value lower than the motion behavior score threshold to obtain a texture feature vector; Based on the structure point coordinates of the device component image segment, a structure response graph is generated, and the structure health score value is calculated based on the texture feature vector and the structure response graph to perform structure state evaluation.

10. The computer vision-based inspection method for highway maintenance construction equipment of claim 9, wherein: The specific steps of calculating the risk level of the device component based on the motion behavior score value and the structure health score value and generating a structured detection report are, Based on the motion behavior score value and the structure health score value, a risk assessment index is constructed, and the risk level is determined through quantile analysis; The motion behavior score value, the structure health score value and the risk level are summarized and arranged to generate a structured detection report.

Images