Fault detection method, device and storage medium

Abstract

Embodiments of the present application provide a fault detection method and device and a storage medium. The method comprises obtaining three-dimensional point cloud data and an optical image of a vehicle to be detected, identifying the optical image to obtain a first position of a target feature point on a wheel of the vehicle to be detected in a first coordinate system, determining a second position of a homonymic point of the target feature point in the three-dimensional point cloud data in a second coordinate system according to the first position and a conversion relationship, determining an edge line of a tail plate of a car of the vehicle to be detected and an edge line of a side plate of the car in the three-dimensional point cloud data according to the second position, and generating a fault reminder message according to the edge line of the tail plate and the edge line of the side plate. The fault detection device provided by the embodiments of the present application can timely issue a fault reminder message to avoid loss. The method does not need to modify the vehicle and is not limited by the performance of the sensor, has high stability, and is safer than manual command.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a fault detection method, device and storage medium. Background Technology

[0002] Dump trucks are construction vehicles, often used in conjunction with excavators and loaders. They are primarily used to transport earth, sand, gravel, and bulk materials for construction projects. The truck body can be automatically raised by hydraulic cylinders; when a load of tens of tons is raised, the vehicle's center of gravity increases, and an accident could result in vehicle destruction and fatalities.

[0003] In related technologies, in-vehicle sensors and other devices can be used to monitor the lifting process of a dump truck.

[0004] However, in the process of realizing this application, the inventors discovered that the prior art has at least the following problems: the sensors are usually not suitable for such heavy-load mechanical devices, and their durability is affected by multiple factors such as harsh environment and pressure. Once the sensor is damaged, the dump truck cannot work properly. Summary of the Invention

[0005] This application provides a fault detection method, device, and storage medium to monitor and provide early warning of faults during the dump truck's bucket lifting process, and to ensure the stability of fault detection.

[0006] In a first aspect, embodiments of this application provide a fault detection method, including:

[0007] Acquire 3D point cloud data and optical images of the vehicle to be inspected;

[0008] The optical image is processed for recognition to obtain the first position of the target feature point on the wheel of the vehicle to be detected in the first coordinate system;

[0009] Based on the first position and the transformation relationship, the second position of the corresponding point of the target feature point in the three-dimensional point cloud data is determined in the second coordinate system; the transformation relationship is the coordinate transformation relationship between the three-dimensional point cloud data and the optical image.

[0010] Based on the second position, the edge line of the tailgate of the vehicle to be detected and the edge line of the side panel of the vehicle are determined in the three-dimensional point cloud data; the edge line of the tailgate is used to connect with the edge line of the side panel when the tailgate is closed.

[0011] A fault alert message is generated based on the edge lines of the tailplate and the sideplate.

[0012] In one possible design, determining the edge line of the rear panel of the vehicle's cabin in the three-dimensional point cloud data based on the second position includes:

[0013] Based on the second position, determine the perpendicular line from the corresponding point to the ground;

[0014] Based on the vertical line, multiple sets of first pixels in the horizontal direction are obtained from the three-dimensional point cloud data; different sets of pixels are on the same horizontal plane; the horizontal direction is perpendicular to the vertical line.

[0015] For each group of first pixels, calculate the distance between each first pixel and the camera position, and determine the first pixel with the shortest distance as the second pixel;

[0016] Based on the positions of multiple second pixels and the optical axis, a first angle between the imaging surface and the side panel of the car is determined;

[0017] Based on the first angle, the edge line of the tail plate in the three-dimensional point cloud data is determined.

[0018] In one possible design, determining the edge line of the tail plate in the three-dimensional point cloud data based on the first angle includes:

[0019] Based on the first angle and multiple second pixels, multiple sets of third pixels are obtained by filtering from multiple sets of first pixels; each set of third pixels is on the same straight line, and the angle between the straight line containing each set of third pixels and the imaging plane is the first angle;

[0020] The edge lines of the side plate and the edge lines of the tail plate are determined based on multiple sets of the third pixel points.

[0021] In one possible design, the method further includes:

[0022] The lifting angle of the car is determined based on multiple sets of third pixel points and multiple second pixel points.

[0023] In one possible design, generating a fault alert message based on the edge lines of the tailplate and the sideplate includes:

[0024] Based on the bucket lifting angle, determine whether the car has entered the bucket lifting state;

[0025] If so, a fault alert message is generated based on the edge lines of the tail plate and the side plate.

[0026] In one possible design, the tailgate is used to rotate and open with its top line as an axis when the car is in the open position; the generation of a fault alert message based on the edge lines of the tailgate and the side plates includes:

[0027] Detect the second angle between the edge of the tailplate and the ground;

[0028] If the second angle is greater than the first threshold, a first fault alert is issued; the first fault alert is used to indicate that the tailgate is in a locked state.

[0029] The area ratio of the point cloud in the region between the edge line of the tail plate and the edge line of the side plate is detected.

[0030] If the area ratio is greater than the second threshold, a second fault alert is issued to cause the vehicle to be inspected to move forward according to the second fault alert; the second fault alert is used to indicate that cargo blockage has occurred during the lifting process.

[0031] In one possible design, the target feature point is the upper or lower vertex of the wheel; the method further includes:

[0032] Based on the second position, determine the first straight line containing the upper and lower vertices of the wheel;

[0033] Determine the third angle between the first straight line and the edge line of the side plate based on the first straight line and the edge line of the side plate;

[0034] Based on the first straight line, determine the fourth angle between the first straight line and the ground;

[0035] If the third angle is greater than the third threshold, or the fourth angle is greater than the fourth threshold, a third fault alert is issued; the third fault alert is used to indicate that the vehicle under test has tilted.

[0036] Secondly, embodiments of this application provide a fault detection device, comprising:

[0037] The acquisition module is used to acquire the 3D point cloud data and optical images of the vehicle to be inspected;

[0038] The processing module is used to perform recognition processing on the optical image to obtain the first position of the target feature point on the wheel of the vehicle to be detected in the first coordinate system;

[0039] The processing module is further configured to determine, based on the first position and the transformation relationship, the second position of the corresponding point of the target feature point in the three-dimensional point cloud data in the second coordinate system; the transformation relationship is the coordinate transformation relationship between the three-dimensional point cloud data and the optical image;

[0040] The processing module is further configured to determine, based on the second position, the edge line of the tailgate of the vehicle to be detected and the edge line of the side panel of the vehicle in the three-dimensional point cloud data; the edge line of the tailgate is configured to connect with the edge line of the side panel when the tailgate is closed.

[0041] The processing module is also used to generate a fault alert message based on the edge lines of the tail plate and the side plate.

[0042] Thirdly, embodiments of this application provide a fault detection device, comprising: at least one processor and a memory;

[0043] The memory stores computer-executed instructions;

[0044] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method described in the first aspect above and various possible designs of the first aspect.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in the first aspect and various possible designs of the first aspect.

[0046] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect and various possible designs of the first aspect.

[0047] The fault detection method, device, and storage medium provided in this embodiment include: acquiring three-dimensional point cloud data and optical images of a vehicle to be detected; performing recognition processing on the optical images to obtain the first position of a target feature point on the wheel of the vehicle to be detected in a first coordinate system; determining the second position of the corresponding point of the target feature point in the three-dimensional point cloud data in a second coordinate system based on the first position and a transformation relationship, wherein the transformation relationship is the coordinate transformation relationship between the three-dimensional point cloud data and the optical image; determining the edge line of the tailgate and the edge line of the side panel of the vehicle to be detected in the three-dimensional point cloud data based on the second position, wherein the edge line of the tailgate is used to connect with the edge line of the side panel when the tailgate is closed; and generating a fault reminder message based on the edge line of the tailgate and the edge line of the side panel. The fault detection device provided in this application acquires both optical images and point cloud data of the vehicle to be detected. Based on the optical images, it determines the first position of the target feature points on the wheels. Then, based on the point cloud data and the first position, it calculates the side panel edge line and the rear panel edge line of the vehicle's cabin. This allows for the analysis of potential dangers of the vehicle based on the side panel edge line and the rear panel edge line, and timely issuance of fault warning messages to avoid losses. This method does not require vehicle modification, is not limited by sensor performance, has high stability, and is safer than manual control. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1a This is a schematic diagram illustrating an application scenario of the fault detection method provided in the embodiments of this application;

[0050] Figure 1b This is a front view schematic diagram of the imaging device provided in the embodiments of this application;

[0051] Figure 2 A schematic flowchart illustrating the fault detection method provided in this application embodiment;

[0052] Figure 3 This is a schematic diagram illustrating the principle of determining multiple second pixel points provided in an embodiment of this application.

[0053] Figure 4 A schematic diagram illustrating the principle of determining the first angle provided in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the bucket-raising state provided in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of the structure of the fault detection device provided in the embodiments of this application;

[0056] Figure 7 This is a structural block diagram of the fault detection device provided in an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] Dump trucks are construction vehicles, often used in conjunction with excavators and loaders. They are primarily used to transport earth, sand, gravel, and bulk materials for construction projects. The truck body can be automatically raised by hydraulic cylinders; when a load of tens of tons is raised, the vehicle's center of gravity increases, and an accident could result in vehicle destruction and fatalities.

[0059] In related technologies, in-vehicle sensors can be used to monitor the dump truck's bucket-lifting process, or personnel can direct it nearby. However, ordinary sensors are not suitable for such heavily loaded mechanical devices, and their durability is affected by harsh environments, pressure, and other factors. Once the sensors fail, the dump truck will not function properly. Using human intervention is strictly prohibited in construction site procedures. In the event of a hydraulic cylinder explosion or vehicle overturning, lives within a certain area would be at risk, constituting a major safety accident.

[0060] To address the aforementioned technical problems, the inventors of this application have discovered that specialized fault detection equipment can be used. This equipment captures images of the vehicle to be inspected (e.g., a dump truck), and based on the captured images, firstly, the positions of target feature points on the wheels are obtained. Then, based on these positions, the positions of the side panel and tail panel edges of the cargo box are calculated. Furthermore, by monitoring the side panel and tail panel edges, the lifting status of the cargo box can be determined, and a fault alert message is issued when a fault is detected. Based on this, the fault detection equipment provided in this application acquires both optical images and point cloud data of the vehicle to be inspected. Based on the optical images, the first positions of target feature points on the wheels are determined. Then, based on the point cloud data and the first positions, the side panel and tail panel edges of the vehicle's cargo box are calculated. This allows for the analysis of potential hazards based on the side panel and tail panel edges, enabling timely issuance of fault alert messages to avoid losses. This method requires no vehicle modification, is not limited by sensor performance, has high stability, and is safer than manual control.

[0061] Figure 1a This is a schematic diagram illustrating an application scenario of the fault detection method provided in the embodiments of this application. Figure 1b This is a front view diagram of the imaging device provided in the embodiments of this application, as shown below. Figure 1a As shown, the fault detection device 110 may include a camera 111 and a controller (not shown, which may be integrated with the camera). Optionally, it may also include a display screen 112, and both the camera 111 and the display screen 112 may be mounted on a bracket 113. The vehicle 70 to be inspected includes a car body 121, which includes wheels 1211, side panels 1212, and a tailgate (not shown). Figure 1b As shown, the shooting device 111 includes a depth camera 1111 and an optical camera 1112, and optionally, may also include a microphone 1113.

[0062] In the specific implementation process, the fault detection device 110 can be set on one side of the vehicle 70 to be inspected. The depth camera 1111 captures point cloud data and sends the point cloud data to the controller. The optical camera captures optical images and sends the optical images to the controller. The controller acquires the three-dimensional point cloud data and optical images of the vehicle to be inspected, inputs the optical images into the recognition model, and obtains the first position of the target feature point on the wheel of the vehicle to be inspected in the first coordinate system. Based on the first position and the transformation relationship, the second position of the corresponding point of the target feature point in the three-dimensional point cloud data in the second coordinate system is determined. The transformation relationship is the coordinate transformation relationship between the three-dimensional point cloud data and the optical image. Based on the second position, the edge line of the tailboard and the edge line of the sideboard of the car body of the vehicle to be inspected are determined in the three-dimensional point cloud data. The edge line of the tailboard is used to connect with the edge line of the sideboard when the tailboard is closed. A fault reminder message is generated based on the edge line of the tailboard and the edge line of the sideboard and displayed on the display screen 11. The driver in the cab of the vehicle under inspection 70 can view the fault warning message displayed on the display screen 112 through the rearview mirror, and then take countermeasures based on the fault warning message. During this process, the microphone can collect the volume intensity at the scene, and the controller can determine whether the bucket lifting state has been entered based on the volume intensity. After entering the bucket lifting state, a fault warning message is generated, and the bucket lifting state can be updated on the display screen 112. For example, if the fault warning indicates that the tailgate is locked, the tailgate can be unlocked. The fault detection device provided in this application embodiment acquires two images of the vehicle under inspection: optical image and point cloud data. Based on the optical image, the first position of the target feature point on the wheel is determined, and then the side panel edge line and tailgate edge line of the vehicle's cabin are calculated based on the point cloud data and the first position. This allows the potential dangers of the vehicle to be analyzed based on the side panel edge line and tailgate edge line, and fault warning messages can be issued in a timely manner to avoid losses. This method does not require vehicle modification, is not limited by sensor performance, has high stability, and is safer than manual command.

[0063] It should be noted that, Figure 1a and Figure 1b The schematic diagram shown is merely an example. The fault detection method and scenario described in this application embodiment are intended to more clearly illustrate the technical solution of this application embodiment and do not constitute a limitation on the technical solution provided in this application embodiment. As those skilled in the art will know, with the evolution of the system and the emergence of new business scenarios, the technical solution provided in this application embodiment is also applicable to similar technical problems.

[0064] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0065] Figure 2 This is a flowchart illustrating the fault detection method provided in an embodiment of this application. Figure 2 As shown, the method includes:

[0066] 201. Acquire the 3D point cloud data and optical images of the vehicle to be inspected.

[0067] The execution entity in this embodiment can be the controller of the fault detection device 110 shown in Figure 1 or the fault detection device 110 itself.

[0068] In this embodiment, a depth camera can be used to acquire 3D point cloud data of the vehicle to be inspected. The depth camera can be a Time-of-Flight (TOF) camera. The TOF camera can be set on the same optical axis as the optical camera and use a lens with the same focal length. Based on this, the acquired point cloud data and optical images can be basically superimposed. TOF technology is a technology that measures the distance between a lens and an object by using the time of flight of a laser. After the laser emitted by the transmitter is received by the receiver, the processor calculates the distance from the lens to the object based on the time of flight of the laser. Multiple transmitters and receivers are combined into an array to acquire a matrix-style point cloud image, that is, 3D point cloud data.

[0069] 202. Perform recognition processing on the optical image to obtain the first position of the target feature point on the wheel of the vehicle to be detected in the first coordinate system.

[0070] Optionally, the optical image can be input into the recognition model to obtain the first position of the target feature point on the wheel of the vehicle to be detected in the first coordinate system. The recognition model can be a neural network model. During the model training process, an AI neural network algorithm can be used to mark the wheel as a positive sample and the rest as negative samples, which are then fed into the model to be trained to identify the position of the wheel.

[0071] In this embodiment, the target feature point can be any feature point on the wheel, such as the upper or lower vertex of the wheel.

[0072] 203. Based on the first position and the transformation relationship, determine the second position of the corresponding point of the target feature point in the three-dimensional point cloud data in the second coordinate system; the transformation relationship is the coordinate transformation relationship between the three-dimensional point cloud data and the optical image.

[0073] Specifically, the optical camera and depth camera can be calibrated in advance to obtain the coordinate transformation relationship between the optical camera coordinate system and the depth camera coordinate system. Thus, after obtaining the first position of the target feature point in the optical camera coordinate system, the second position of the target feature point in the point cloud data can be obtained using this coordinate transformation relationship.

[0074] 204. Based on the second position, determine the edge line of the tailgate of the vehicle to be detected and the edge line of the side panel of the vehicle in the three-dimensional point cloud data; the edge line of the tailgate is used to connect with the edge line of the side panel when the tailgate is closed.

[0075] 205. Generate a fault alert message based on the edge lines of the tail plate and the side plate.

[0076] Optionally, determining the edge line of the tailgate of the vehicle to be detected in the three-dimensional point cloud data based on the second position may include: determining the perpendicular line from the corresponding point to the ground based on the second position; obtaining multiple sets of first pixels in the horizontal direction from the three-dimensional point cloud data based on the perpendicular line; different sets of pixels are on the same horizontal plane; the horizontal direction is perpendicular to the perpendicular line; for each set of first pixels, calculating the distance between each first pixel and the camera position, and determining the first pixel with the shortest distance as the second pixel; determining the first angle between the imaging surface and the side panel of the vehicle based on the multiple second pixels and the optical axis position; and determining the edge line of the tailgate in the three-dimensional point cloud data based on the first angle.

[0077] In some embodiments, determining the edge line of the tail plate in the three-dimensional point cloud data based on the first angle may include: selecting multiple sets of third pixels from multiple sets of first pixels based on the first angle and multiple second pixels; each set of third pixels is on the same straight line, and the angle between the straight line containing each set of third pixels and the imaging surface is the first angle; determining the edge line of the side plate and the edge line of the tail plate based on the multiple sets of third pixels.

[0078] Specifically, such as Figure 3As shown, after obtaining the second position of the target feature point of the wheel in the 3D point cloud data, taking the target feature point as the vertex on the wheel as an example, starting from the vertex on the wheel, a straight line search is performed along the horizontal axis in the 3D point cloud data at certain intervals. This yields multiple sets of data Dp, i.e., multiple sets of first pixel points. Each set of Dp can include pixels from the side of the dump truck's cargo box 121, as well as pixels from other parts on the same horizontal plane, such as the dump truck's cab. In each set of data Dp, the shortest data set Ds from the depth camera 111 to the cargo box 121 is searched, i.e., multiple second pixel points, along the vertical axis from top to bottom or from bottom to top. The shortest data should fit onto a straight line, and this is done for all Ds points. Figure 4 As shown, since ∠1=∠2, and ∠2+∠3=90°, ∠3+∠A=90°, then ∠A=∠2. Based on the distance from depth camera 111 to pixel O and the distance from depth camera 111 to pixel Ds, the angle of ∠2 can be determined, and thus the angle of ∠A can be obtained, which is the first angle.

[0079] After obtaining the first angle A, for each group Dp, a line segment can be fitted, centered on the shortest point Ds and extending to the left and right. The length L of each line segment is calculated, and multiple line segments L1, L2, ..., Ln are fitted from top to bottom, which are multiple groups of third pixels. The third pixel of each group is the first pixel in each Dp array on the straight line ∠A. At the right end of L1, L2, ..., Ln, since the tail plate is perpendicular to the side plate, the distance between the pixels on the tail plate and the depth camera will increase. A straight line is fitted at the inflection point of the distance increase. This straight line is the position Dw of the edge line connecting the tail plate and the side plate, that is, the position of the edge line of the side plate and the edge line of the tail plate. It should be perpendicular to the ground.

[0080] Optionally, the method may further include: determining the lifting angle of the car based on multiple sets of the third pixel points and multiple sets of the second pixel points.

[0081] For example, a microphone can be used to sample the audio at the scene and record the sound intensity Sq at a certain moment. Since the dump truck will make a lot of noise when it starts to lift the bucket, when a sudden change in Sq is detected, it can be determined that the vehicle under test has entered the lifting state.

[0082] like Figure 5As shown, a depth camera continuously acquires 3D point cloud data, and a controller continuously detects the 3D point cloud data, constantly forming new Dsn arrays. During the detection process upward with the wheel vertex as the axis, as the truck bed rises, the line segment corresponding to Dsn will show a decrease in length. Z1 and Z0 are the distances between Ds1 and Ds0 on the left side of the truck bed. The angle ∠B of the raised truck bed can be determined from the distance relationship between Z1 and Z0. Optionally, multiple Dsn can be selected to determine ∠B. When the angles determined by multiple Dsn are all approximately ∠B, it is determined that the truck bed has entered the raised state, and the angle is ∠B. This can be combined with the method of determining the raised truck bed based on the microphone to confirm that the truck bed is in the raised state.

[0083] During the bucket lifting process, the tailgate of the car 121 will be vertically downward under the influence of gravity. In the 3D point cloud data, the position of the tailgate should remain vertically downward, and the newly detected Dwn angle should remain within a certain threshold. If a change occurs, it may be due to the latch failing to unlock, causing the tailgate to fail to open. Based on this, a first fault warning message can be generated, indicating a danger and requiring the bucket lifting to be stopped on the display screen.

[0084] After the bucket is lifted, there will be a certain angle between the side panel line of the tracking car and the tail panel line of the car. As the cargo is tilted, objects can be detected at this angle, such as... Figure 5 As shown, the angle between Dw (i.e., the side panel edge line) and the new Dwn (the rear panel edge line after the bucket is lifted) can be tracked. Within a certain range below, that is, in the area with Dwn as the edge and an angle of ∠B, the TOF camera tracks the number of point clouds in this area. When the stacked material fills the angled portion, that is, when the proportion of the point cloud area in the triangular area of ​​the angle exceeds a threshold, a second fault warning message can be generated to remind the driver to move the vehicle forward. After the vehicle moves, the wheels can be searched based on the optical image, and the original position of the car body 121 and the rear panel in the three-dimensional point cloud data can be translated to above the wheels to continue the fault detection for lifting the bucket.

[0085] Optionally, generating a fault alert message based on the edge lines of the tailboard and the sideboard may include: determining whether the car has entered the lifting state based on the lifting angle; if so, generating a fault alert message based on the edge lines of the tailboard and the sideboard.

[0086] Optionally, the tailgate is used to rotate and open with its top line as an axis when the car is in the lifting state; the generation of a fault reminder message based on the edge lines of the tailgate and the side plates may include: detecting a second angle between the edge line of the tailgate and the ground; if the second angle is greater than a first threshold, issuing a first fault reminder; the first fault reminder is used to indicate that the tailgate is in a locked state; detecting the area ratio of the point cloud in the region between the edge line of the tailgate and the edge line of the side plates; if the area ratio is greater than a second threshold, issuing a second fault reminder to cause the vehicle to be tested to move forward according to the second fault reminder; the second fault reminder is used to indicate that cargo blockage has occurred during the lifting process.

[0087] The fault detection method provided in this embodiment acquires both optical images and point cloud data of the vehicle to be detected. Based on the optical images, it determines the first position of the target feature points on the wheels. Then, based on the point cloud data and the first position, it calculates the side panel edge line and the rear panel edge line of the vehicle's cabin. This allows for the analysis of potential dangers of the vehicle based on the side panel edge line and the rear panel edge line, and timely issuance of fault warning messages to avoid losses. This method does not require vehicle modification, is not limited by sensor performance, has high stability, and is safer than manual control.

[0088] In some embodiments, the target feature point is the upper or lower vertex of the wheel; the method may further include: determining a first straight line containing the upper and lower vertices of the wheel based on the second position; determining a third angle between the first straight line and the edge line of the side plate based on the first straight line and the edge line of the side plate; determining a fourth angle between the first straight line and the ground based on the first straight line; issuing a third fault alert if the third angle is greater than a third threshold or the fourth angle is greater than a fourth threshold; the third fault alert is used to indicate that the vehicle under test is tilted.

[0089] Specifically, the position of the wheel is searched in the 3D point cloud data, and the upper and lower vertices are connected to obtain a straight line Lt. A perpendicular line Lc is taken from the plane where the car is located. The angle between Lt and Lc, as well as the angle between Lt and the ground, should be kept within the corresponding threshold range during the lifting process. If the angle exceeds the threshold range, a third fault reminder message can be generated and displayed on the display screen to indicate that the car is tilted and prompt to stop lifting the car.

[0090] In practice, the driver uses the display screen to move the vehicle and raise the bucket, reducing the risk of uncertainty about the vehicle's condition during the bucket-raising process. Furthermore, the display screen uses colored icons to represent operating instructions, improving visibility.

[0091] Figure 6This is a schematic diagram of the structure of the fault detection device provided in an embodiment of this application. Figure 6 As shown, the fault detection device 60 includes an acquisition module 601 and a processing module 602.

[0092] The acquisition module 601 is used to acquire the three-dimensional point cloud data and optical image of the vehicle to be inspected;

[0093] Processing module 602 is used to perform recognition processing on the optical image to obtain the first position of the target feature point on the wheel of the vehicle to be detected in the first coordinate system;

[0094] The processing module 602 is further configured to determine, based on the first position and the transformation relationship, the second position of the corresponding point of the target feature point in the three-dimensional point cloud data in the second coordinate system; the transformation relationship is the coordinate transformation relationship between the three-dimensional point cloud data and the optical image;

[0095] The processing module 602 is further configured to determine, based on the second position, the edge line of the tailgate of the vehicle to be detected and the edge line of the side panel of the vehicle in the three-dimensional point cloud data; the edge line of the tailgate is configured to connect with the edge line of the side panel when the tailgate is closed.

[0096] The processing module 602 is also used to generate a fault reminder message based on the edge line of the tail plate and the edge line of the side plate.

[0097] The fault detection device provided in this application acquires both optical images and point cloud data of the vehicle to be detected. Based on the optical images, it determines the first position of the target feature points on the wheels. Then, based on the point cloud data and the first position, it calculates the side panel edge line and the rear panel edge line of the vehicle's cabin. This allows for the analysis of potential dangers of the vehicle based on the side panel edge line and the rear panel edge line, and timely issuance of fault warning messages to avoid losses. This method does not require vehicle modification, is not limited by sensor performance, has high stability, and is safer than manual control.

[0098] The fault detection device provided in this application embodiment can be used to execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0099] Figure 7 This is a structural block diagram of a fault detection device provided in an embodiment of this application. The device may be a computer, a tablet, or the like.

[0100] The device 70 may include one or more of the following components: a processing component 701, a memory 702, a power supply component 703, a multimedia component 704, an audio component 705, an input / output (I / O) interface 706, a sensor component 707, and a communication component 708.

[0101] Processing component 701 typically controls the overall operation of device 70, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 701 may include one or more processors 709 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 701 may include one or more modules to facilitate interaction between processing component 701 and other components. For example, processing component 701 may include a multimedia module to facilitate interaction between multimedia component 704 and processing component 701.

[0102] Memory 702 is configured to store various types of data to support the operation of device 70. Examples of such data include instructions for any application or method operating on device 70, contact data, phonebook data, messages, pictures, videos, etc. Memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0103] Power supply assembly 703 provides power to the various components of device 70. Power supply assembly 703 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 70.

[0104] Multimedia component 704 includes a screen that provides an output interface between the device 70 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 704 includes a front-facing camera and / or a rear-facing camera. When the device 70 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0105] Audio component 705 is configured to output and / or input audio signals. For example, audio component 705 includes a microphone (MIC) configured to receive external audio signals when device 70 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 702 or transmitted via communication component 708. In some embodiments, audio component 705 also includes a speaker for outputting audio signals.

[0106] I / O interface 706 provides an interface between processing component 701 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0107] Sensor assembly 707 includes one or more sensors for providing state assessments of various aspects of device 70. For example, sensor assembly 707 can detect the on / off state of device 70, the relative positioning of components such as the display and keypad of device 70, changes in position of device 70 or a component of device 70, the presence or absence of user contact with device 70, orientation or acceleration / deceleration of device 70, and temperature changes of device 70. Sensor assembly 707 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 707 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications, and a depth camera for acquiring three-dimensional point cloud data. In some embodiments, sensor assembly 707 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0108] Communication component 708 is configured to facilitate wired or wireless communication between device 70 and other devices. Device 70 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 708 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 708 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0109] In an exemplary embodiment, device 70 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0110] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 702 including instructions, which can be executed by a processor 709 of the device 70 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0111] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0112] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0113] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0114] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the fault detection method executed by the fault detection device described above.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A fault detection method, characterized in that, include: Acquire 3D point cloud data and optical images of the vehicle to be inspected; The optical image is processed for recognition to obtain the first position of the target feature point on the wheel of the vehicle to be detected in the first coordinate system; Based on the first position and the transformation relationship, determine the second position of the corresponding point of the target feature point in the three-dimensional point cloud data in the second coordinate system; The transformation relationship is the coordinate transformation relationship between the three-dimensional point cloud data and the optical image; Based on the second position, determine the perpendicular line from the corresponding point to the ground; Based on the vertical line, multiple sets of first pixels in the horizontal direction are obtained from the three-dimensional point cloud data; different sets of pixels are on the same horizontal plane; the horizontal direction is perpendicular to the vertical line. For each group of first pixels, calculate the distance between each first pixel and the camera position, and determine the first pixel with the shortest distance as the second pixel; Based on the positions of multiple second pixels and the optical axis, a first angle between the imaging surface and the side panel of the car is determined; Based on the first angle, the edge line of the tail plate in the three-dimensional point cloud data is determined; the edge line of the tail plate is used to connect with the edge line of the side plate when the tail plate is closed. A fault alert message is generated based on the edge lines of the tailplate and the sideplate, including: Detect the second angle between the edge of the tailplate and the ground; If the second angle is greater than the first threshold, a first fault alert is issued; the first fault alert is used to indicate that the tailgate is in a locked state. The area ratio of the point cloud in the region between the edge line of the tail plate and the edge line of the side plate is detected. If the area ratio is greater than the second threshold, a second fault alert is issued to cause the vehicle to be inspected to move forward according to the second fault alert; the second fault alert is used to indicate that cargo blockage has occurred during the lifting process.

2. The method according to claim 1, characterized in that, Determining the edge line of the tail plate in the three-dimensional point cloud data based on the first angle includes: Based on the first angle and multiple second pixels, multiple sets of third pixels are obtained by filtering from multiple sets of first pixels; each set of third pixels is on the same straight line, and the angle between the straight line containing each set of third pixels and the imaging plane is the first angle; The edge lines of the side plate and the edge lines of the tail plate are determined based on multiple sets of the third pixel points.

3. The method according to claim 2, characterized in that, The method further includes: The lifting angle of the car is determined based on multiple sets of third pixel points and multiple second pixel points.

4. The method according to claim 3, characterized in that, The step of generating a fault alert message based on the edge lines of the tail plate and the side plate includes: Based on the bucket lifting angle, determine whether the car has entered the bucket lifting state; If so, a fault alert message is generated based on the edge lines of the tail plate and the side plate.

5. The method according to any one of claims 1-4, characterized in that, The tailgate is used to rotate and open with the top line of the tailgate as the axis when the car is in the lifting state.

6. The method according to any one of claims 1-4, characterized in that, The target feature point is the upper or lower vertex of the wheel; the method further includes: Based on the second position, determine the first straight line containing the upper and lower vertices of the wheel; Determine the third angle between the first straight line and the edge line of the side plate based on the first straight line and the edge line of the side plate; Based on the first straight line, determine the fourth angle between the first straight line and the ground; If the third angle is greater than the third threshold, or the fourth angle is greater than the fourth threshold, a third fault alert is issued; the third fault alert is used to indicate that the vehicle under test has tilted.

7. A fault detection device, characterized in that, include: The acquisition module is used to acquire the 3D point cloud data and optical images of the vehicle to be inspected; The processing module is used to perform recognition processing on the optical image to obtain the first position of the target feature point on the wheel of the vehicle to be detected in the first coordinate system; The processing module is further configured to determine, based on the first position and the transformation relationship, the second position of the corresponding point of the target feature point in the three-dimensional point cloud data in the second coordinate system; the transformation relationship is the coordinate transformation relationship between the three-dimensional point cloud data and the optical image; The processing module is also configured to determine the perpendicular line from the corresponding point to the ground based on the second position; Based on the vertical line, multiple sets of first pixels in the horizontal direction are obtained from the three-dimensional point cloud data; different sets of pixels are on the same horizontal plane; the horizontal direction is perpendicular to the vertical line. For each group of first pixels, calculate the distance between each first pixel and the camera position, and determine the first pixel with the shortest distance as the second pixel; Based on the positions of multiple second pixels and the optical axis, a first angle between the imaging surface and the side panel of the car is determined; Based on the first angle, the edge line of the tail plate in the three-dimensional point cloud data is determined; the edge line of the tail plate is used to connect with the edge line of the side plate when the tail plate is closed. The processing module is also used to generate a fault reminder message based on the edge line of the tail plate and the edge line of the side plate; The processing module is specifically used to detect the second angle between the edge of the tailplate and the ground; If the second angle is greater than the first threshold, a first fault alert is issued; the first fault alert is used to indicate that the tailgate is in a locked state. The area ratio of the point cloud in the region between the edge line of the tail plate and the edge line of the side plate is detected. If the area ratio is greater than the second threshold, a second fault alert is issued to cause the vehicle to be inspected to move forward according to the second fault alert; the second fault alert is used to indicate that cargo blockage has occurred during the lifting process.

8. A fault detection device, characterized in that, include: An optical camera, a depth camera, at least one processor, and memory; Both the optical camera and the depth camera are connected to the processor. The optical camera is used to take pictures of the vehicle to be inspected, obtain optical images, and send the optical images to the processor; The depth camera is used to photograph the vehicle to be detected, obtain three-dimensional point cloud data, and send the three-dimensional point cloud data to the processor; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the fault detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the fault detection method as described in any one of claims 1 to 6.

Images