A control system and method based on unmanned road patrol tow truck

Abstract

The present invention provides a control system and method based on an unmanned road patrol tow truck, relating to the field of automatic tow removal technology. The control system comprises a multimodal perception module composed of an image sensor, a lidar, and a depth sensor, which acquires two-dimensional image data, three-dimensional point cloud data, and three-dimensional depth data. A hierarchical detection module combining a dual-stream deep neural network with a self-attention mechanism fuses data features and outputs a multi-level score, enabling accurate vehicle damage assessment and intelligent selection of tow removal modes. The system is continuously optimized through adaptive learning. Through multi-sensor collaboration, dual-stream deep neural network feature extraction, and a dynamic self-attention fusion mechanism, the present invention achieves high-precision vehicle status analysis and intelligent tow removal decision-making, enabling intelligent and accurate judgment of the damage status of vehicles experiencing road accidents, reducing the risk of secondary damage caused by damage assessment errors, reducing the frequency of manual intervention, and significantly improving the efficiency of road tow removal operations.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic obstacle removal, and in particular to a control system and method based on an unmanned road patrol obstacle removal vehicle. Background Art

[0002] With the rapid development of artificial intelligence, autonomous driving, and smart transportation technologies, intelligent and automated road management systems centered around unmanned vehicles are gaining increasing attention. Among them, unmanned road patrol tow trucks, capable of autonomous patrols, timely detection of roadside vehicles, and rapid and effective disposal, have become a key development direction for intelligent transportation and smart cities. Tow trucks typically choose a towing method based on the damage to the vehicle. For example, for a minor tow truck, the tow truck tows the vehicle to a support structure using a tow hook. For a severely damaged tow truck, the tow truck connects to the vehicle's four wheels via a lifting mechanism and hoists it to a support structure. Alternatively, the tow truck lifts and secures one end of the vehicle through a lifting mechanism and then directly tows it for towing.

[0003] In recent years, researchers have proposed a number of automated or semi-automated tow truck removal solutions that rely on cameras or lidar to identify vehicles through optical visual recognition of their exterior features. However, current unmanned tow truck systems generally suffer from low recognition accuracy and limited data processing methods. This makes it difficult to accurately and effectively assess vehicle damage in complex environments, making it difficult to avoid manual intervention during actual tow truck removal operations. Furthermore, due to the lack of intelligence in data processing and decision-making, traditional unmanned tow trucks are unable to accurately distinguish between appropriate towing methods (such as towing, lifting, or hoisting) for different degrees of damage. This leads to frequent misjudgments, which reduces tow truck efficiency and can even cause secondary damage accidents.

[0004] Therefore, in order to meet the higher requirements of modern traffic management for efficiency, intelligence and reliability of obstacle removal operations, it is necessary to develop an unmanned obstacle removal control system and method with more accurate damage assessment capabilities, more flexible operation decision-making methods and more intelligent data fusion processing mechanisms. Summary of the Invention

[0005] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to propose a control system and method based on an unmanned road patrol tow truck, which adopts the following technical solutions:

[0006] On one hand, the present invention provides a control system based on an unmanned road patrol tow truck, comprising:

[0007] The multimodal perception module includes at least:

[0008] Image sensor, used to obtain two-dimensional image data of the faulty vehicle and extract vehicle body identification information through OCR;

[0009] A laser sensor for generating three-dimensional point cloud data of the appearance of the faulty vehicle;

[0010] A depth sensor for generating three-dimensional depth data of the local appearance of the faulty vehicle;

[0011] Hierarchical detection module, including a deep neural network computing model with a two-stream architecture:

[0012] The first-stream network uses a convolutional neural network to extract the image feature vector F of the above two-dimensional image data img , output the first confidence score C1;

[0013] The second-stream network uses a point cloud processing algorithm to extract the point cloud feature vector F of the above three-dimensional point cloud data. pc And the depth feature vector F of the three-dimensional depth data dep ; Superimpose and fuse the above-mentioned two-dimensional image data and three-dimensional point cloud data, and output a second confidence score C2; ​​if the above-mentioned second confidence score C2 ≤ 85, then update and fuse the above-mentioned three-dimensional point cloud data and three-dimensional depth data, and output a third confidence score C3;

[0014] Feature fusion unit, which uses self-attention mechanism to weightedly fuse the above image feature vector F img , point cloud feature vector F pc And the depth feature vector F dep , generate fusion feature vector F fusion ;

[0015] The control processing unit is connected to the above-mentioned hierarchical detection module and selects the obstacle removal mode according to the preset threshold rule. When the above-mentioned third confidence score C3 is less than or equal to 85, manual intervention is requested.

[0016] As a further improvement, the image sensor scans the front and rear of the faulty vehicle, and the calculation formula of the first confidence score C1 is:

[0017] ;

[0018] Among them, F img The image feature vector representing the output of the first-stream convolutional neural network;

[0019] W1 is the weight matrix, b1 is the bias term;

[0020] σ(·) is a Sigmoid activation function that maps the first confidence score C1 to the range of 0~100.

[0021] Scan the front and rear of the faulty vehicle. Usually, the front and rear of the vehicle are marked with information such as the vehicle brand logo and vehicle model. The image sensor determines the damage to the front and rear of the vehicle on the one hand, and collects the vehicle information of the faulty vehicle through OCR on the other hand.

[0022] As a further improvement, the laser sensor scans both sides and the roof of the faulty vehicle, and the calculation formula of the second confidence score C2 is:

[0023] ;

[0024] Among them, W2 is the weight matrix and b2 is the bias term;

[0025] σ(·) is the Sigmoid activation function, which maps the second confidence score C2 to the range of 0~100;

[0026] Fusion feature vector F fusion(img,pc) The above two-dimensional image data and three-dimensional point cloud data are fused through the self-attention mechanism. The specific formula is:

[0027] ;

[0028] The above weight coefficient c 1 and c 2 Satisfaction c 1+ c 2=1, and when the first confidence score C1>75, that is, when the front and rear of the faulty vehicle are less damaged, reduce the weight coefficient c 1. Increase the weight coefficient c 2.

[0029] As a further improvement, the depth sensor scans the wheel and axle area of ​​the faulty vehicle, and the calculation formula of the third confidence score C3 is:

[0030] ;

[0031] Among them, W3 is the weight matrix and b3 is the bias term;

[0032] σ(·) is the Sigmoid activation function, which maps the third confidence score C3 to the range of 0~100;

[0033] Fusion feature vector F fusion(pc,dep) The above two-dimensional image data and three-dimensional point cloud data are fused through the self-attention mechanism. The specific formula is:

[0034] ;

[0035] Wherein, M is a mask matrix, and the overlapping part of the above-mentioned three-dimensional point cloud data and the above-mentioned three-dimensional depth data is updated to the above-mentioned three-dimensional depth data through the mask matrix M;

[0036] The above weight coefficient c 2´ and c 3 Satisfaction c 2´+ c 3=1, and the weight coefficient c 3> c 2´.

[0037] As a further improvement, it also includes an adaptive learning module, which is connected to the above-mentioned control processing unit and optimizes the weight coefficients in the above-mentioned feature fusion unit through offline learning and incremental learning.

[0038] As a further improvement, the offline learning of the above weight coefficients adopts the loss function of minimizing the mean square error:

[0039] ;

[0040] Among them, F target,i It is the standard feature vector of the vehicle state marked after manual intervention.

[0041] For further improvement, the incremental learning method of the above weight coefficient is:

[0042] When the recent system detection error shows a continuous increasing trend, the weight coefficient is updated using the following rules:

[0043] ;

[0044] Accuracy is the percentage of tasks in the last five tasks where the error between the system and the test score and the manual verification score is within ±5;

[0045] Δ is the learning rate, initially limited to 0.01~0.05;

[0046] When Accuracy ≥ 95%, the weight coefficient update is stopped to maintain the parameter stability of the system.

[0047] As a further improvement, the preset rule for the control processing unit to select the obstacle removal mode is:

[0048] When the first confidence score C1 is greater than 75 and the second confidence score C2 is greater than 85, the control executes the towing mode to tow the faulty vehicle at a speed not exceeding 20 km / h;

[0049] When the first confidence score C1 is greater than 75 and the second confidence score C2 is less than or equal to 85, if the third confidence score C3 is greater than 85, the lifting mode is executed; if the third confidence score C3 is less than or equal to 85, manual intervention is requested;

[0050] When the first confidence score C1 is less than or equal to 75 and the second confidence score C2 is greater than or equal to 85, the control executes the hoisting mode;

[0051] When the first confidence score C1 ≤ 75 and the second confidence score C2 ≤ 85, if the third confidence score C3 > 85, the lifting mode is executed; if the third confidence score C3 ≤ 85, manual intervention is requested.

[0052] Another aspect of the present invention provides a control method based on an unmanned road patrol tow truck, which is applied to the control system proposed in any one of the above items, comprising the following steps:

[0053] S10: Control the image sensor to collect two-dimensional image data of the front and rear of the faulty vehicle, and extract the image feature vector F of the two-dimensional image data through the first-stream network. img , and generate a first confidence score C1;

[0054] S20: Control the laser sensor to obtain three-dimensional point cloud data of the two sides and roof area of ​​the faulty vehicle, and use the second stream network to extract the point cloud feature vector F of the three-dimensional point cloud data. pc ;

[0055] S21: superimpose and fuse the above-mentioned two-dimensional image data and three-dimensional point cloud data through the self-attention mechanism, and generate a second confidence score C2;

[0056] S30: Select the corresponding obstacle removal mode based on the scoring result:

[0057] When the first confidence score C1 is greater than 75 and the second confidence score C2 is greater than 85, the towing mode is selected to perform the towing operation by connecting the tow hook of the faulty vehicle;

[0058] When the first confidence score C1 is less than or equal to 75, and the second confidence score C2 is greater than or equal to 85, the hoisting mode is selected to lift the faulty vehicle by connecting the four wheels and perform towing operations;

[0059] S40: If the second confidence score C2 ≤ 85, control the depth sensor to obtain three-dimensional depth data of the wheel and axle area of ​​the faulty vehicle, and use the second flow network to extract the depth feature vector F of the three-dimensional depth data dep ;

[0060] S41: The three-dimensional point cloud data and the three-dimensional depth data are updated and fused through a self-attention mechanism, and a third confidence score C3 is generated;

[0061] S50: Select the corresponding obstacle removal mode based on the scoring results:

[0062] When the third confidence score C3 is greater than 85, the lifting mode is selected to lift and fix one end of the faulty vehicle and tow the faulty vehicle for towing.

[0063] When the third confidence score C3≤85, request manual intervention;

[0064] S60: Records the scoring results of each task and actual clearance feedback in real time, and optimizes the model weight coefficients through online incremental learning and offline batch learning to improve the accuracy of subsequent detection tasks.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] First, the technical solution of the present invention uses a multimodal perception module composed of image sensors, laser sensors and depth sensors to comprehensively perceive and collect data on faulty vehicles, and jointly constructs a multidimensional representation of vehicle damage with two-dimensional images, three-dimensional point clouds and three-dimensional depth data. The multimodal data acquisition method enables the system to overcome the problem of one-sided information recognition by a single sensor. Specifically, the image sensor obtains the front and rear damage status of the vehicle body and vehicle identification information, and uses the first confidence score C1 to determine whether the towing conditions are met. The laser sensor further obtains three-dimensional point cloud data of the vehicle body excluding the front and rear, and superimposes it with the two-dimensional image data to form a second confidence score C2 that represents the damage status of the entire vehicle body. If the towing conditions are not met, if the second confidence score C2 is higher than the preset value, that is, the vehicle body damage is minor, the four wheels of the faulty vehicle are connected through the lifting mechanism and the vehicle is lifted for towing. Furthermore, when the second confidence score C2 is lower than the preset value, it indicates that the vehicle body is severely damaged. At this time, if lifting is used, the vehicle may overturn due to imbalance of the vehicle body. Therefore, the wheel and axle area are further scanned by the depth sensor, and the third confidence score C3 is output to determine whether the wheel can rotate. If it can, the lifting mode is used for towing. That is, one end of the faulty vehicle is lifted and fixed, and towed and moved. The above technical solution effectively improves the accuracy and comprehensiveness of the assessment of damage to faulty vehicles. By judging the damage to the vehicle and selecting the appropriate clearance method, it significantly reduces the probability of misjudgment while effectively avoiding the risk of secondary damage caused by wrong decisions, thereby improving the overall operational efficiency and safety of traffic management, and has obvious practical application advantages.

[0067] Secondly, the technical solution of the present invention includes a hierarchical detection module based on a dual-stream architecture, which processes multiple modal data separately through a deep neural network to achieve an efficient feature extraction and feature fusion process. When the second confidence score C2 is not enough to provide a clear judgment, the system will update the fused three-dimensional depth data, replace the three-dimensional depth data representing the wheels and axles with the original three-dimensional point cloud data, and generate a more refined third confidence score C3 to enhance the damage identification accuracy of the wheel area. The misjudgment of the operation mode due to insufficient damage assessment is effectively avoided through a step-by-step refinement evaluation method. In the face of minor damage, the depth sensor is not called, which saves system resources, greatly improves the system's adaptability to complex scenarios, and improves the accuracy of decision-making and the reliability of obstacle clearance operation selection.

[0068] Third, the technical solution of the present invention also includes an adaptive learning module to achieve dynamic optimization of the weight coefficient. After manual intervention, the processing results are entered into the system, a standard feature vector is generated, and the weight coefficient is corrected through offline learning. On the other hand, when the deviation between the last five scoring results and manual verification is too large, the weight coefficient is corrected through incremental learning. The adaptive learning module effectively enhances the robustness and environmental adaptability of the system, ensuring the high precision and stability of the unmanned tow truck control system in long-term operation under actual complex road conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 It is a framework diagram of the system of the present invention;

[0071] Figure 2 is a flow chart of the steps of the method of the present invention;

[0072] Figure 3 This is a flow chart of the obstacle removal mode selection rules in the present invention. DETAILED DESCRIPTION

[0073] In order to facilitate understanding by those skilled in the art, the structure of the present invention is further described in detail with reference to the embodiments and the accompanying drawings:

[0074] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Terms such as "portion," "side," and "end" indicate positions or relationships based on those shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They do not indicate or imply that the devices or components indicated must have, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0075] like Figure 1 As shown, the present application provides a control system based on an unmanned road patrol tow truck, including a multimodal perception module, a hierarchical detection module and a control processing unit.

[0076] like Figure 1 As shown, the multimodal perception module includes at least an image sensor, a laser sensor, and a depth sensor. The image sensor is used to acquire 2D image data of the faulty vehicle and extract vehicle identification information through optical character recognition (OCR). Specifically, it is an optical imaging device such as a high-resolution camera, capable of clearly capturing the faulty vehicle's license plate number, vehicle model, and surface damage details. The laser sensor is used to generate 3D point cloud data of the faulty vehicle's exterior. Specifically, it is a device such as a 3D lidar, which actively emits laser pulses and measures reflection time to obtain 3D point cloud data of the target object's surface. Lidar has high distance and angle measurement accuracy and is unaffected by ambient lighting variations. It can efficiently and stably acquire 3D structural data of the faulty vehicle's exterior and damaged areas. The depth sensor, specifically a depth imaging device such as a structured light scanner or time-of-flight camera, is used to generate 3D depth data of the faulty vehicle's partial exterior. It calculates target depth information by deforming light, making it suitable for accurately capturing detailed damage information on the vehicle's chassis, wheels, and axles. This facilitates further assessment of the vehicle's mobility and avoids unsuccessful clearance operations due to missed underbody damage. The above-mentioned image sensors, laser sensors and depth sensors are controlled by the movement of the tow truck body and the body robotic arm to scan the front, rear, left and right sides, roof and other areas of the faulty vehicle, and align the data through spatiotemporal calibration. The data representation error is ≤2cm, so that subsequent data fusion is accurate.

[0077] Further, such as Figure 1 As shown in the figure, the layered detection module includes a deep neural network computing model with a two-stream architecture. Specifically, the first-stream network adopts a convolutional neural network, such as the ResNet model or the DenseNet model, which gradually extracts the image feature vector F from the input two-dimensional image data through multi-layer convolution operations. img , forming a first confidence score C1. This is used to determine the extent of damage to the front and rear of the faulty vehicle and to determine whether the faulty vehicle meets the towing conditions.

[0078] The second-stream network uses a point cloud processing neural network model based on PointNet or PointNet++ to fuse and extract features from the 3D point cloud data obtained by the laser sensor and the 3D depth data obtained by the depth sensor. The MLP structure inside the neural network performs feature mapping, clustering, and dimensionality reduction on each point cloud data point to obtain the point cloud feature vector F of the 3D point cloud data. pc And the depth feature vector F of the three-dimensional depth data dep . The two-dimensional image data and the three-dimensional point cloud data are superimposed and fused, and a second confidence score C2 is output. In one embodiment, the two-dimensional image data represents the damage information on the front and rear of the vehicle body, and the three-dimensional point cloud data represents the damage information on the sides and roof of the vehicle body. The superposition and fusion of the two represent the damage information on the exterior of the vehicle body. If the second confidence score C2 ≤ 85, the three-dimensional point cloud data and the three-dimensional depth data are updated and fused, and a third confidence score C3 is output to further determine the degree of damage to the faulty vehicle and the applicable clearance method. Specifically, the three-dimensional depth data represents the local information of the wheels and axles, which overlaps with part of the representation information of the three-dimensional point cloud data. Therefore, the three-dimensional depth data is updated to the three-dimensional point cloud data, that is, the two are updated and fused to form new three-dimensional data, which is used to further determine the degree of damage to the wheels of the faulty vehicle.

[0079] Furthermore, the hierarchical detection module also includes a feature fusion unit. The feature fusion unit adopts a self-attention mechanism based on the Transformer architecture. When fusing feature vectors of multiple data modalities, this mechanism can automatically generate an attention weight coefficient based on the importance of each modal feature to highlight key damage information. Through the feature fusion process, the negative impact of missing or misjudgment of single modal information can be further reduced, and the accuracy and robustness of the overall vehicle damage assessment can be improved. Specifically, the self-attention mechanism is used to weight the fusion of the image feature vector F img , point cloud feature vector F pc And the depth feature vector F dep , generate fusion feature vector F fusion .

[0080] like Figure 1 As shown, the control processing unit is connected to the hierarchical detection module and selects the obstacle removal mode according to the preset threshold rule. When the third confidence score C3≤85, manual intervention is requested.

[0081] like Figure 3 As shown, the preset rules for the control processing unit to select the obstacle removal mode are specifically:

[0082] When the first confidence score C1 > 75 and the second confidence score C2 > 85, the control executes the towing mode to tow the faulty vehicle at a speed not exceeding 20 km / h.

[0083] In this rule, when the first confidence score C1 is greater than 75, it is judged that the damage to the front and rear of the vehicle is relatively minor, meeting the conditions for installing a tow hook. Therefore, the damage to the vehicle body is further judged. If the second confidence score C2 is greater than 85, it is determined that the towing mode can be executed.

[0084] When the first confidence score C1 ≤ 75 and the second confidence score C2 > 85, the control executes the hoisting mode.

[0085] In this rule, when the first confidence score C1 ≤ 75, it means that the front and rear of the vehicle are severely damaged and do not meet the installation conditions of the tow hook. Therefore, the damage to the vehicle body is further judged. If the second confidence score C2 is greater than 85, it means that the damage to the vehicle body is relatively light. The lifting mode is executed. The faulty vehicle is fixed to the wheel area through the lifting mechanism, lifted and placed on the load-bearing part of the tow truck for transportation.

[0086] When the first confidence score C1 is greater than 75 and the second confidence score C2 is less than or equal to 85, the control executes the lifting mode if the third confidence score C3 is greater than or equal to 85, and requests manual intervention if the third confidence score C3 is less than or equal to 85.

[0087] When the first confidence score C1 ≤ 75 and the second confidence score C2 ≤ 85, if the third confidence score C3 > 85, the control executes the lifting mode; if the third confidence score C3 ≤ 85, manual intervention is requested.

[0088] In this rule, if the second confidence score C2 ≤ 85, it indicates significant vehicle damage. The damage could be to the vehicle or the wheels. If the vehicle is severely damaged, the imbalance of the vehicle could cause the vehicle to overturn during lifting, resulting in secondary damage. If the wheels are severely damaged, towing could result in unpredictable risks due to wheel loss of steering or directional deviation. Therefore, a third confidence score C3 is required for further analysis. If the third confidence score C3 > 85, the wheel damage is minor (the first and second confidence scores C1 and C2 are not considered in this case). Lifting is recommended for towing purposes. This involves lifting one end of the vehicle and supporting the other end on the wheels, with the tow truck towing the vehicle. If the third confidence score C3 ≤ 85, the wheel damage is significant, requiring manual intervention and employing more complex towing methods, such as lifting the vehicle chassis or manually securing the vehicle for transport.

[0089] In one embodiment, a tow truck uses information such as the vehicle brand, vehicle model, and vehicle identification code identified by an image sensor to move the faulty vehicle to a maintenance site or temporary storage point on a preset route, thereby completing an automated tow removal task.

[0090] The present invention proposes an unmanned tow truck control system based on multimodal perception and hierarchical confidence assessment. Through multi-sensor collaboration, dual-stream deep neural network feature extraction and self-attention dynamic fusion mechanism, high-precision vehicle status analysis and intelligent tow clearance decision-making are achieved, and intelligent and accurate judgment of the damage status of vehicles with road faults is realized. It can effectively reduce the risk of secondary damage caused by damage assessment errors and reduce the frequency of manual intervention. It has the advantages of high degree of automation, strong stability, and strong environmental adaptability. It can significantly improve the efficiency, safety and intelligence level of road towing operations, and has broad practical application value and good market prospects.

[0091] The working principles of the hierarchical detection module and the adaptive learning module are described below through a specific embodiment:

[0092] The image sensor scans the front and rear of the faulty vehicle. The first confidence score C1 is calculated as follows:

[0093] ;

[0094] Among them, F img Represents the image feature vector output by the first-stream convolutional neural network, W1 is the weight matrix, and F img is a vector of the same dimension, used to convert F img Mapped to the scoring space, b1 is the bias term with an initial value of 0; σ(·) is the Sigmoid activation function, whose original linear output mapping interval is [0,1]. It is amplified by 100 times to map the first confidence score C1 to the range of 0~100.

[0095] Image feature vector F img The calculation formula is:

[0096] ;

[0097] Among them, Q img , K img , V img are the query matrix Q, key matrix K, and value matrix V in the self-attention mechanism, generated from two-dimensional image data.

[0098] The laser sensor scans both sides and the roof of the faulty vehicle. The second confidence score C2 is calculated as follows:

[0099] ;

[0100] Where W2 is the weight matrix, b2 is the bias term, and σ(·) is the Sigmoid activation function, which is amplified by 100 times to map the second confidence score C2 to the range of 0~100.

[0101] Fusion feature vector Ffusion(img,pc) It is generated by fusing two-dimensional image data and three-dimensional point cloud data through the self-attention mechanism. The specific formula is:

[0102] ;

[0103] Among them, Q img , K img , V img The query matrix Q, key matrix K, and value matrix V in the self-attention mechanism are generated from two-dimensional image data; Q pc , K pc , V pc The query matrix Q, key matrix K, and value matrix V are generated from 3D point cloud data.

[0104] Weight coefficient c 1 and c 2 Satisfaction c 1+ c 2=1, and when the first confidence score C1>75, that is, when the front and rear of the faulty vehicle are less damaged, reduce the weight coefficient c 1. Increase the weight coefficient c 2. In a preferred embodiment, c The initial value of 1 is 0.5, c The initial value of 2 is 0.5. If C1>75, then c 1 changes to 0.3, c 2 changes to 0.7.

[0105] The depth sensor scans the wheel and axle area of ​​the faulty vehicle, and the third confidence score C3 is calculated as follows:

[0106] ;

[0107] Where W3 is the weight matrix, b3 is the bias term, and σ(·) is the Sigmoid activation function, which is amplified by 100 times to map the third confidence score C3 to the range of 0~100.

[0108] Fusion feature vector F fusion(pc,dep) It is generated by fusing two-dimensional image data and three-dimensional point cloud data through the self-attention mechanism. The specific formula is:

[0109] ;

[0110] Where M is a mask matrix, and the overlapping portion of the 3D point cloud data and the 3D depth data is updated to 3D depth data using the mask matrix M. In this embodiment, the overlapping portion is the side surface of the wheel and the axle. In another embodiment, the depth sensor is also used to detect the vehicle chassis, in which case the 3D depth data of the vehicle chassis is not included in the update of the 3D point cloud data.

[0111] Weight coefficient c 2´ and c 3 Satisfaction c 2´+ c 3=1, and the weight coefficient c 3> c 2´, as a preference, c 2´=0.3, c 3=0.7.

[0112] The system also includes an adaptive learning module connected to the control processing unit, which optimizes the weight coefficients in the feature fusion unit through offline learning and incremental learning.

[0113] The offline learning of weight coefficients adopts the loss function of minimizing the mean square error:

[0114] ;

[0115] Among them, F target,i It is the standard feature vector of the vehicle state marked after manual intervention. It can be the standard feature vector converted from historical average data or the standard feature vector manually calibrated by professionals. Through N groups of data, the fusion feature vector F is forward calculated. fusion（·） , update the above-mentioned weight matrix, weight parameters and other learnable variables.

[0116] The incremental learning method of weight coefficients is:

[0117] When the recent system detection error shows a continuous increasing trend, the weight coefficient is updated using the following rules:

[0118] ;

[0119] Accuracy is the percentage of tasks in the last five tasks where the error between the system and the test score and the manual verification score is within ±5;

[0120] Δ is the learning rate, initially limited to 0.01~0.05;

[0121] When Accuracy ≥ 95%, the weight coefficient update is stopped to maintain the parameter stability of the system.

[0122] In some embodiments, the learning rate can be dynamically adjusted with the accuracy, for example:

[0123] If Accuracy = 93%, Δ is reduced to 0.01, stabilizing the system;

[0124] If Accuracy = 70%, Δ is increased to 0.05 to accelerate convergence.

[0125] like Figure 2 As shown, another aspect of the present invention provides a control method based on an unmanned road patrol tow truck, which is applied to the above-mentioned control system and includes the following steps:

[0126] S10: Control the image sensor to collect two-dimensional image data of the front and rear of the faulty vehicle, and extract the image feature vector F of the two-dimensional image data through the first-stream network. img , and generate a first confidence score C1;

[0127] S20: Control the laser sensor to obtain the three-dimensional point cloud data of the two sides and roof area of ​​the faulty vehicle, and use the second stream network to extract the point cloud feature vector F of the three-dimensional point cloud data pc ;

[0128] S21: The two-dimensional image data and the three-dimensional point cloud data are superimposed and fused through the self-attention mechanism, and a second confidence score C2 is generated;

[0129] S30: Select the corresponding obstacle removal mode based on the scoring result:

[0130] When the first confidence score C1 is greater than 75 and the second confidence score C2 is greater than 85, the towing mode is selected to perform the towing operation by connecting the tow hook of the faulty vehicle;

[0131] When the first confidence score C1 is less than or equal to 75, and the second confidence score C2 is greater than or equal to 85, the hoisting mode is selected to lift the faulty vehicle by connecting the four wheels and perform towing operations;

[0132] S40: If the second confidence score C2 ≤ 85, control the depth sensor to obtain three-dimensional depth data of the wheel and axle area of ​​the faulty vehicle, and use the second flow network to extract the depth feature vector F of the three-dimensional depth data dep ;

[0133] S41: The 3D point cloud data and the 3D depth data are updated and fused through the self-attention mechanism, and a third confidence score C3 is generated;

[0134] S50: Select the corresponding obstacle removal mode based on the scoring results:

[0135] When the third confidence score C3 is greater than 85, the lifting mode is selected to lift and fix one end of the faulty vehicle and tow the faulty vehicle for towing.

[0136] When the third confidence score C3≤85, request manual intervention;

[0137] S60: Records the scoring results of each task and actual clearance feedback in real time, and optimizes the model weight coefficients through online incremental learning and offline batch learning to improve the accuracy of subsequent detection tasks.

[0138] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A control system based on an unmanned road patrol tow truck, characterized in that: include: The multimodal perception module includes at least: Image sensor, used to obtain two-dimensional image data of the faulty vehicle and extract vehicle body identification information through OCR; A laser sensor for generating three-dimensional point cloud data of the appearance of the faulty vehicle; A depth sensor for generating three-dimensional depth data of the local appearance of the faulty vehicle; Hierarchical detection module, including a deep neural network computing model with a two-stream architecture: The first stream network uses a convolutional neural network to extract the image feature vector F of the two-dimensional image data img , output the first confidence score C1; The second stream network uses a point cloud processing algorithm to extract the point cloud feature vector F of the three-dimensional point cloud data. pc And the depth feature vector F of the three-dimensional depth data dep ; Superimpose and fuse the two-dimensional image data and the three-dimensional point cloud data, and output a second confidence score C2; ​​if the second confidence score C2 ≤ 85, update and fuse the three-dimensional point cloud data and the three-dimensional depth data, and output a third confidence score C3; Feature fusion unit, which uses self-attention mechanism to weightedly fuse the image feature vector F img , point cloud feature vector F pc And the depth feature vector F dep , generate fusion feature vector F fusion ; The control processing unit is connected to the hierarchical detection module and selects an obstacle removal mode according to a preset threshold rule, and requests manual intervention when the third confidence score C3 is less than or equal to 85.

2. A control system based on an unmanned road patrol tow truck as claimed in claim 1, characterized in that: The image sensor scans the front and rear of the faulty vehicle, and the calculation formula of the first confidence score C1 is: ； Among them, F img The image feature vector representing the output of the first-stream convolutional neural network; W1 is the weight matrix, b1 is the bias term; σ(·) is a Sigmoid activation function, which maps the first confidence score C1 to the range of 0~100.

3. A control system based on an unmanned road patrol tow truck as claimed in claim 2, characterized in that: The laser sensor scans both sides of the vehicle body and the roof of the faulty vehicle. The calculation formula of the second confidence score C2 is: ； Among them, W2 is the weight matrix and b2 is the bias term; σ(·) is a Sigmoid activation function that maps the second confidence score C2 to the range of 0 to 100; Fusion feature vector F fusion(img,pc) The two-dimensional image data and the three-dimensional point cloud data are fused through the self-attention mechanism. The specific formula is: ； The weight coefficient γ 1 and γ 2 Satisfaction γ 1+ γ 2=1, and when the first confidence score C1>75, that is, when the front and rear of the faulty vehicle are less damaged, reduce the weight coefficient γ 1. Increase the weight coefficient γ 2.

4. A control system based on an unmanned road patrol tow truck as claimed in claim 3, characterized in that: The depth sensor scans the wheel and axle area of ​​the faulty vehicle, and the calculation formula of the third confidence score C3 is: ； Among them, W3 is the weight matrix and b3 is the bias term; σ(·) is a Sigmoid activation function that maps the third confidence score C3 to the range of 0 to 100; Fusion feature vector F fusion(pc,dep) The two-dimensional image data and the three-dimensional point cloud data are fused through the self-attention mechanism. The specific formula is: ； Wherein, M is a mask matrix, and the overlapping parts of the 3D point cloud data and the 3D depth data are updated to the 3D depth data through the mask matrix M. Since the laser sensor scans the sides and top of the vehicle body, and the depth sensor scans the wheels and axles, there is a certain overlap in the scanning areas of the two. Therefore, a mask matrix M with the same dimension as the eigenvector of the above data is introduced, in which the elements are 0 or 1. Specifically defined as: when an element of the mask matrix M is 1, it means that the corresponding position is scanned by both the laser sensor and the depth sensor, and the position is replaced by the 3D depth data of the depth sensor. The weight coefficient γ 2´ and γ 3 Satisfaction γ 2´+ γ 3=1, and the weight coefficient γ 3> γ 2´.

5. The control system based on an unmanned road patrol tow truck according to claim 1, characterized in that: It also includes an adaptive learning module, which is connected to the control processing unit and optimizes the weight coefficients in the feature fusion unit through offline learning and incremental learning.

6. A control system based on an unmanned road patrol tow truck as claimed in claim 5, characterized in that: The offline learning of the weight coefficients adopts the loss function of minimizing the mean square error: ； Among them, F target,i It is the standard feature vector of the vehicle state marked after manual intervention.

7. A control system based on an unmanned road patrol tow truck as claimed in claim 6, characterized in that: The incremental learning method of the weight coefficient is: When the recent system detection error shows a continuous increasing trend, the weight coefficient is updated using the following rules: ； Accuracy is the percentage of tasks in the last five tasks where the error between the system and the test score and the manual verification score is within ±5; Δ is the learning rate, initially limited to 0.01~0.05; When Accuracy ≥ 95%, the weight coefficient update is stopped to maintain the parameter stability of the system.

8. The control system based on an unmanned road patrol tow truck according to claim 1, characterized in that: The preset rule for the control processing unit to select the obstacle removal mode is: When the first confidence score C1 is greater than 75 and the second confidence score C2 is greater than 85, the control executes the towing mode to tow the faulty vehicle at a speed not exceeding 20 km / h; When the first confidence score C1 is greater than 75 and the second confidence score C2 is less than or equal to 85, if the third confidence score C3 is greater than 85, the lifting mode is executed; if the third confidence score C3 is less than or equal to 85, manual intervention is requested; When the first confidence score C1 is less than or equal to 75 and the second confidence score C2 is greater than or equal to 85, the control executes the hoisting mode; When the first confidence score C1 ≤ 75 and the second confidence score C2 ≤ 85, if the third confidence score C3 > 85, the lifting mode is executed; if the third confidence score C3 ≤ 85, manual intervention is requested.

9. A control method based on an unmanned road patrol tow truck, applied to the control system according to any one of claims 1 to 8, characterized in that: The steps include: S10: Control the image sensor to collect two-dimensional image data of the front and rear of the faulty vehicle, and extract the image feature vector F of the two-dimensional image data through the first-stream network. img , and generate a first confidence score C1; S20: Control the laser sensor to obtain three-dimensional point cloud data of the two sides and roof area of ​​the faulty vehicle, and use the second flow network to extract the point cloud feature vector F of the three-dimensional point cloud data pc ; S21: superimposing and fusing the two-dimensional image data and the three-dimensional point cloud data through a self-attention mechanism, and generating a second confidence score C2; S30: Select the corresponding obstacle removal mode based on the scoring result: When the first confidence score C1 is greater than 75 and the second confidence score C2 is greater than 85, the towing mode is selected to perform the towing operation by connecting the tow hook of the faulty vehicle; When the first confidence score C1 is less than or equal to 75, and the second confidence score C2 is greater than or equal to 85, the hoisting mode is selected to lift the faulty vehicle by connecting the four wheels and perform towing operations; S40: If the second confidence score C2 ≤ 85, control the depth sensor to obtain three-dimensional depth data of the wheel and axle area of ​​the faulty vehicle, and use the second flow network to extract the depth feature vector F of the three-dimensional depth data dep ; S41: updating and fusing the three-dimensional point cloud data and the three-dimensional depth data through a self-attention mechanism, and generating a third confidence score C3; S50: Select the corresponding obstacle removal mode based on the scoring results: When the third confidence score C3 is greater than 85, the lifting mode is selected to lift and fix one end of the faulty vehicle and tow the faulty vehicle for towing. When the third confidence score C3≤85, request manual intervention; S60: Records the scoring results of each task and actual clearance feedback in real time, and optimizes the model weight coefficients through online incremental learning and offline batch learning to improve the accuracy of subsequent detection tasks.

Images