Vehicle weld defect detection method and device, program product and electronic equipment

Abstract

The invention relates to the technical field of vehicle detection, and provides a vehicle weld defect detection method and device, a program product and electronic equipment. The method comprises the following steps: acquiring a to-be-detected vehicle image corresponding to a to-be-detected vehicle, inputting the to-be-detected vehicle image into a pre-trained welding seam defect detection model, and obtaining a welding seam type, a welding seam type and a welding seam defect type corresponding to a welding seam in the to-be-detected vehicle according to the output of the welding seam defect detection model; and based on the welding seam type, the welding seam type and the welding seam defect type corresponding to the welding seam, determining the defect grade of the welding seam according to the defect size of the welding seam defect and a pre-trained welding seam defect grade division model. According to the scheme, the accuracy and efficiency of vehicle weld defect detection can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle inspection technology, and more specifically, to a method for detecting defects in vehicle welds, a device for detecting defects in vehicle welds, a computer program product, and an electronic device. Background Technology

[0002] Mining dump trucks are now widely used in open-pit mines. The main components, such as the cargo box, frame, and walkway, are made up of a large number of welded parts. Detecting weld defects in the welded parts of the vehicle can improve the production quality of the vehicle and thus improve the safety of the vehicle in the future.

[0003] In related technologies, each weld is mainly inspected manually based on experience. However, the types and varieties of welds in welded parts are diverse, and weld defects are usually hidden, resulting in a high rate of false detection and missed detection in manual inspection, as well as low inspection efficiency.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method and apparatus for detecting defects in vehicle welds, a computer program product, and an electronic device, thereby improving the accuracy and efficiency of vehicle weld defect detection to at least a certain extent.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to a first aspect of this disclosure, a method for detecting weld defects in vehicles is provided, comprising: acquiring an image of a vehicle to be inspected corresponding to the vehicle to be inspected; inputting the image of the vehicle to be inspected into a pre-trained weld defect detection model; obtaining the weld type, weld type, and weld defect type corresponding to the weld in the vehicle to be inspected based on the output of the weld defect detection model; and determining the defect level of the weld based on the weld type, weld type, and weld defect type corresponding to the weld, according to the defect size of the weld defect and a pre-trained weld defect level classification model.

[0008] According to a second aspect of this disclosure, a vehicle weld defect detection device is provided, comprising: a defect detection module configured to acquire an image of a vehicle to be inspected, input the image of the vehicle to be inspected into a pre-trained weld defect detection model, and obtain the weld type, weld type, and weld defect type corresponding to the weld in the image of the vehicle to be inspected based on the output of the weld defect detection model; and a defect level determination module configured to determine the defect level of the weld based on the weld type, weld type, and weld defect type corresponding to the weld, according to the defect size of the weld defect and a pre-trained weld defect level classification model.

[0009] According to a third aspect of this disclosure, a computer program product comprising instructions is provided, which, when run on a computer, causes the computer to perform the steps of the vehicle weld defect detection method as described in the first aspect.

[0010] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the vehicle weld defect detection method as described in the first aspect of the above embodiments.

[0011] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the vehicle weld defect detection method as described in the first aspect of the above embodiments.

[0012] As can be seen from the above technical solutions, the vehicle weld defect detection method, the vehicle weld defect detection device, and the computer program product and electronic device for implementing the vehicle weld defect detection method in the exemplary embodiments of this disclosure have at least the following advantages and positive effects: In the technical solutions provided by some embodiments of this disclosure, firstly, by acquiring an image of the vehicle to be inspected, the image is input into a pre-trained weld defect detection model. The weld type, weld type, and weld defect type of the weld in the vehicle to be inspected can be obtained based on the output of the weld defect detection model. Then, based on the weld type, weld type, and weld defect type, the defect level of the weld can be determined according to the defect size of the weld defect and the pre-trained weld defect level classification model. Compared with related technologies, this disclosure has several advantages. First, it enables automatic detection of weld defects through a pre-trained weld defect detection model, improving the efficiency of weld defect detection and avoiding the problems of false detection and missed detection caused by manual inspection due to its inability to handle large-scale inspection demands, thus improving the accuracy of weld defect detection. Second, this disclosure allows for the quantitative assessment of weld defects through weld defect levels, providing more accurate data support for quality control in the vehicle manufacturing process and helping to improve vehicle production quality. Third, by determining the weld defect levels, this disclosure helps relevant personnel to devote more energy to weld defects that require repair, thereby improving the efficiency of weld repair.

[0013] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0015] Figure 1 This diagram illustrates a process flow of a vehicle weld defect detection method according to an exemplary embodiment of the present disclosure. Figure 2 This illustration shows a flowchart of a method for obtaining the weld type, weld type, and weld defect type corresponding to a weld in a vehicle to be inspected, according to an exemplary embodiment of this disclosure. Figure 3 A flowchart illustrating a method for determining a true weld seam according to an exemplary embodiment of this disclosure is shown. Figure 4 A flowchart illustrating a method for determining the level of weld defects according to an exemplary embodiment of this disclosure is shown. Figure 5 This diagram illustrates a weld defect detection system according to an exemplary embodiment of the present disclosure; Figure 6 This diagram illustrates the composition of a vehicle weld defect detection device according to an exemplary embodiment of the present disclosure. Figure 7 A schematic diagram of the structure of an electronic device in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0016] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0017] The terms “a,” “an,” “the,” and “the” are used in this specification to indicate the presence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first” and “second” are used only as markings and are not a limitation on the number of objects.

[0018] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0019] In related technologies, vehicle weld defects are usually detected by visual inspection or tapping. This method is inefficient, relies on human experience, and has low accuracy and reliability.

[0020] Figure 1 This diagram illustrates a flowchart of a vehicle weld defect detection method according to an exemplary embodiment of this disclosure, with reference to... Figure 1 The method includes: Step S110: Collect the image of the vehicle to be inspected corresponding to the vehicle to be inspected, input the image of the vehicle to be inspected into the pre-trained weld defect detection model, and obtain the weld type, weld type and weld defect type corresponding to the weld in the vehicle to be inspected based on the output of the weld defect detection model. Step S120: Based on the weld type, weld type, and weld defect type corresponding to the weld, determine the defect level of the weld according to the defect size of the weld defect and the pre-trained weld defect level classification model.

[0021] exist Figure 1 In the technical solution provided by the embodiment shown, firstly, by acquiring the image of the vehicle to be inspected, the image is input into a pre-trained weld defect detection model. The weld type, weld type, and weld defect type of the weld in the vehicle to be inspected can be obtained according to the output of the weld defect detection model. Then, based on the weld type, weld type, and weld defect type, the defect level of the weld can be determined according to the defect size of the weld defect and the pre-trained weld defect level classification model. Compared with related technologies, this disclosure has several advantages. First, it enables automatic detection of weld defects through a pre-trained weld defect detection model, improving the efficiency of weld defect detection and avoiding the problems of false detection and missed detection caused by manual inspection due to its inability to handle large-scale inspection demands, thus improving the accuracy of weld defect detection. Second, this disclosure allows for the quantitative assessment of weld defects through weld defect levels, providing more accurate data support for quality control in the vehicle manufacturing process and helping to improve vehicle production quality. Third, by determining the weld defect levels, this disclosure helps relevant personnel to devote more energy to weld defects that require repair, thereby improving the efficiency of weld repair.

[0022] The following is a detailed description of the specific implementation method of "step S110, acquiring the image of the vehicle to be detected corresponding to the vehicle to be detected, inputting the image of the vehicle to be detected into the pre-trained weld defect detection model, and obtaining the weld type, weld type and weld defect type corresponding to the weld in the vehicle to be detected based on the output of the weld defect detection model".

[0023] In one exemplary embodiment, the vehicle to be inspected in this disclosure may include a vehicle manufactured in a factory. For example, the method in this disclosure can automatically detect and evaluate weld defects in a vehicle before it leaves the factory.

[0024] In one exemplary implementation, a two-dimensional image or a three-dimensional point cloud image corresponding to the vehicle to be detected can be acquired.

[0025] For example, after vehicle production is complete, industrial cameras and micro-drones can be used to capture images of the entire vehicle, thus obtaining a two-dimensional image of the vehicle to be inspected. Alternatively, multiple industrial cameras or micro-drones can simultaneously acquire vehicle images from different angles, and then these images can be stitched together to obtain a two-dimensional image of the vehicle to be inspected. Three-dimensional point cloud images of the vehicle to be inspected can be acquired using LiDAR or structured light 3D scanners. In these cases, the two-dimensional image and the three-dimensional point cloud image are acquired simultaneously.

[0026] For example, Figure 2 This diagram illustrates a process flow of a method for obtaining the weld type, weld subtype, and weld defect type corresponding to a weld in a vehicle to be inspected, according to an exemplary embodiment of this disclosure. (See reference...) Figure 2 The method may include steps S210 to S250. Wherein: In step S210, the two-dimensional image is input into the pre-trained weld recognition model, and the candidate welds in the two-dimensional image are obtained based on the output of the two-dimensional image.

[0027] For example, images of welded parts can be pre-acquired, and the weld seam area can be marked in the images to obtain a first sample dataset. An arbitrary object detection model can then be trained using this first sample dataset to obtain a pre-trained weld seam recognition model. In other words, the pre-trained weld seam recognition model can identify the weld seam area in an image, thereby obtaining candidate weld seams.

[0028] In step S220, the two-dimensional coordinates of the candidate weld in the two-dimensional image are converted into three-dimensional spatial coordinates through a pre-calibrated coordinate mapping model.

[0029] For example, a relationship between pixel coordinates and 3D spatial coordinates can be established through camera calibration parameters, thereby obtaining a pre-calibrated coordinate mapping model. Then, based on this coordinate influence model, for each candidate weld, the 2D coordinates of the candidate weld determined by the weld recognition model can be converted into 3D spatial coordinates.

[0030] In step S230, the point cloud data corresponding to the candidate weld is determined from the three-dimensional point cloud image based on the three-dimensional spatial coordinates.

[0031] For example, a subset of point clouds whose coordinates fall within the three-dimensional spatial coordinate region of a candidate weld can be selected from a three-dimensional point cloud image to obtain the point cloud data corresponding to the candidate weld.

[0032] In step S240, the actual weld is determined from the candidate welds based on the point cloud data.

[0033] For example, Figure 3This diagram illustrates a flowchart of a method for determining the actual weld seam according to an exemplary embodiment of this disclosure. (Reference) Figure 3 The method may include steps S310 to S350. Wherein: In step S310, the normal vectors of the points in the point cloud data of the candidate weld are calculated.

[0034] For example, the normal vector of each point in the point cloud data of a candidate weld can be calculated. For instance, for each point, its k nearest neighbors can be taken, and a local plane can be fitted using the least squares method. The normal vector of the plane is the normal vector of that point.

[0035] In step S320, the angle between the normal vectors of adjacent points is determined, and the first number of determined angles is counted.

[0036] For example, the angle between the normal vectors of each pair of adjacent points in the point cloud data of the candidate weld can be determined, and the first number of determined angles can be counted.

[0037] In step S330, the included angles less than the threshold are determined as target included angles, and the first ratio between the second number of target included angles and the first number is calculated.

[0038] The threshold can be customized based on experience. The threshold is used to measure the consistency of the direction of the normal vectors of adjacent points, so the threshold needs to be set relatively small. If a certain angle is smaller than the threshold, the angle can be determined as the target angle. Then, the second number of target angles is counted, and the first ratio between the second number and the first number is calculated.

[0039] In step S340, the actual weld is determined from the candidate welds based on the first ratio.

[0040] The first ratio is used to measure the consistency of the direction of the normal vectors of adjacent points in the point cloud data of the candidate weld. The first ratio can be determined according to the requirements, such as 0.7.

[0041] For example, if the first ratio is greater than the first preset value, the candidate weld can be identified as a suspected weld.

[0042] As a special area formed after welding, such as a protrusion or depression, the normal vectors of adjacent points on the surface of a weld tend to be in the same direction. Using a first ratio, candidate welds with discontinuous changes in the normal vectors of adjacent points can be filtered out, and the remaining candidate welds can be classified as suspected welds.

[0043] In step S350, for the suspected weld, the change curve of the included angle is fitted, and the actual weld is determined from the suspected weld based on the change curve of the included angle.

[0044] For example, if the curve of the angle change indicates that the angle change between adjacent points shows a linear and gradual pattern, the suspected weld is identified as the real weld.

[0045] The weld is a continuous physical structure with a smooth transition in surface morphology. The normal vectors of its surface points tend to be consistent locally, and the overall change is regular without abrupt changes. Step S350 can further filter out suspected welds with irregular changes in the angle between adjacent normal vectors, thus identifying the remaining suspected welds as genuine welds.

[0046] Through the steps S310 to S350 described above, two-dimensional images and three-dimensional point cloud images can be combined to improve the accuracy of weld seam identification in the vehicle under inspection and avoid identifying other vehicle parts as weld seams.

[0047] In step S250, based on the two-dimensional image and the position of the actual weld in the two-dimensional image, the weld type, weld type, and weld defect type corresponding to the weld in the vehicle to be detected are obtained based on the output of the pre-trained weld defect detection model.

[0048] For example, a training dataset can be generated in advance, and a machine learning model can be trained based on the training dataset to obtain a pre-trained weld recognition model.

[0049] For example, various welded component images can be pre-collected. The weld location, type, and presence of defects can be labeled on these images. If defects are found, their types can be further labeled, thus obtaining sample data labels. The sample dataset is then divided into training and testing datasets. An initial machine learning model is iteratively trained based on the training dataset and its corresponding labels until the training loss value is less than a preset value or the preset number of iterations is reached. Training then stops, yielding the target machine learning model. The target machine learning model is then tested using the testing dataset. If the performance test is passed, the target machine learning model is designated as the pre-trained weld defect detection model.

[0050] In one exemplary embodiment, the weld defect detection model may consist of a convolutional neural network sub-model and a target detection sub-model. The target detection sub-model may include the YOLOv8 model, but may also include other models; this exemplary embodiment does not impose any specific limitations on this.

[0051] For example, the input data can be first fed into a convolutional neural network sub-model, which extracts weld features from the input data. Then, the weld features extracted by the convolutional neural network sub-model are fed into a YOLOv8 model, which processes the weld features. The detection result of weld defects can be obtained based on the output of the YOLOv8 model.

[0052] The convolutional neural network sub-model can automatically extract weld features without the need for manual feature design, reducing the time cost of manual feature design and further improving the efficiency of weld defect detection.

[0053] In one exemplary embodiment, the weld type may include pipe weld, plate weld, T-type fillet weld, etc., the weld type may include spot weld, incomplete weld, full weld, etc., and the weld defect type may include crack, porosity, lack of fusion, etc.

[0054] For example, one exemplary implementation of step S250 may include: cropping a real weld image from a two-dimensional image based on the position of the real weld in the two-dimensional image of the vehicle to be inspected; enlarging the real weld image to obtain a target real weld image corresponding to the real weld image; inputting the target real weld image and the cooling temperature of the real weld into a pre-trained weld defect detection model, and obtaining the weld type, weld type and weld defect type corresponding to the weld in the vehicle to be inspected based on the output of the weld defect detection model.

[0055] For example, the output of a pre-trained weld recognition model can include the two-dimensional coordinate position of each candidate weld in a two-dimensional image. The weld position can be represented by a bounding box, and its position can be determined using the two-dimensional center coordinates of the bounding box and the two-dimensional coordinates of its four vertices (top left, bottom left, top right, and bottom right). For the actual weld among the candidate welds, the actual weld image can be cropped from the two-dimensional image based on its position within the vehicle's two-dimensional graphic. This image is then enlarged to obtain the target actual weld image. This target actual weld image is input into a pre-trained weld defect detection model. The model's output determines the weld type, weld type, presence of defects, and, if present, the type of defect.

[0056] By cropping and enlarging real weld images before inputting them into the weld defect detection model, the model can extract more detailed features from the weld, improving its accuracy in weld defect detection. Furthermore, cropping reduces interference from background information unrelated to the weld, thus increasing the model's detection efficiency.

[0057] The following is a detailed description of the specific implementation method of "step S120, determining the defect level of the weld based on the weld type, weld type, and weld defect type corresponding to the weld, according to the defect size of the weld defect and the pre-trained weld defect level classification model".

[0058] For example, Figure 4 This diagram illustrates a flowchart of a method for determining the level of weld defects according to an exemplary embodiment of this disclosure. (See reference...) Figure 4 The method may include steps S410 to S460. Wherein: In step S410, the defect size corresponding to the weld defect in the weld is calculated.

[0059] In one exemplary embodiment, weld defects can be pre-quantified to obtain weld defect dimensions.

[0060] For example, for crack-type defects, the defect size can be calculated based on the detected defect location rectangle. For instance, the defect size corresponding to a crack-type defect can be obtained using the following formula (1).

[0061] (1)

[0062] In formula (1), This indicates the position of the leftmost pixel column in the crack area (i.e., the minimum value at the left edge of the crack). This indicates the position of the topmost pixel row in the crack region (i.e., the minimum value at the top edge of the crack). This indicates the position of the rightmost pixel column in the crack area (i.e., the maximum value at the right edge of the crack). This indicates the position of the bottom row of pixels in the crack area (i.e., the maximum value at the bottom edge of the crack).

[0063] For example, for pore-type defects, the defect size can be obtained by the pore area. For example, the defect size corresponding to the pore-type defect can be obtained by the following formula (2).

[0064] (2)

[0065] In formula (2), The width of the rectangle representing the location of the defect.

[0066] In step S420, the weld type, weld defect type and defect size corresponding to the weld are input into a pre-trained weld defect level classification model, and the first defect level of the weld is obtained according to the output of the weld defect level classification model.

[0067] For example, training data can be extracted in advance from documents such as audio and video lectures by experts, enterprise standards for steel structure welding, and international standards. Specifically, weld types, weld defect types, defect sizes, and defect levels can be extracted from these documents as sample datasets, and then a weld defect level classification model can be trained based on this sample dataset.

[0068] In actual testing, after obtaining the weld type, weld defect type, and defect size of the weld in the vehicle to be tested, these data can be input into the weld defect level classification model, and the first defect level of the weld can be obtained based on the output of the model.

[0069] The weld defect level model can divide weld defects into two levels, such as level 1 and level 2. Level 2 defects are more severe than level 2 defects. For example, level 1 indicates fatal and irreparable defects, while level 2 indicates repairable defects.

[0070] In step S430, the second defect level corresponding to the defect size of the weld is determined according to the preset mapping relationship between the weld defect size threshold and the defect level.

[0071] In one exemplary embodiment, various literature materials can be sorted out to obtain a preset mapping relationship between weld defect size thresholds and defect levels corresponding to different types and types of welds.

[0072] For example, according to a certain industry standard, a defect level of more than 3 mm in length can be classified as Level 1, meaning an unrepairable defect. A defect level of less than 1 mm in diameter and less than 3 pores can be classified as Level 2, meaning a repairable defect, thus establishing a mapping relationship.

[0073] Then, during the actual inspection process, the calculated defect size can be matched with the weld defect size in the mapping relationship, and the defect level indicated by the successfully matched weld defect size threshold can be determined as the second defect level corresponding to the calculated weld defect size.

[0074] In step S440, it is determined whether the first defect level and the second defect level are the same. If they are the same, proceed to step S450; otherwise, proceed to step S460.

[0075] In step S450, the first defect level or the second defect level is determined as the defect level of the weld.

[0076] For example, if the first defect level and the second defect level are the same, the first defect level or the second defect level can be directly determined as the defect level of the weld.

[0077] In step S460, the degree of difference between the defect size and the defect size threshold of the second defect level is obtained, as well as the output confidence of the weld defect level classification model. Based on the degree of difference and the output confidence, the target defect level is determined from the first defect level and the second defect level, and the target defect level is determined as the defect level of the weld.

[0078] For example, if the first defect level and the second defect level are not the same, it means that one of them has lower reliability. Therefore, the reliability of the two can be evaluated, and the one with higher reliability can be selected as the final defect level of the weld.

[0079] For example, for the second defect level, the distance D between the defect size and the nearest weld defect size threshold can be calculated. The smaller D is, the closer the defect is to the threshold edge, and the lower the reliability of the second defect level. For the first defect level, the confidence level can be output using methods such as Monte Carlo. The distance D and the confidence level can be normalized, and the weld with the highest reliability can be selected as the final defect level based on the normalization result.

[0080] Through steps S410 to S460 described above, the first defect level and the second defect level can be mutually verified to improve the reliability and accuracy of defect level determination.

[0081] For example, as mentioned above, the vehicle to be inspected in this disclosure includes vehicles manufactured in a factory. Based on this, the vehicle weld defect detection method in this disclosure may further include: dividing the vehicle image to be inspected into regions according to vehicle element types to obtain sub-images to be inspected corresponding to each vehicle element type, wherein the vehicle element type includes one or more of a cargo box, a cab, and a frame; calculating the weld defect rate in each sub-image to be inspected in each vehicle image to be inspected; for each sub-image to be inspected, obtaining the target weld defect rate of the vehicle element type indicated by the sub-image to be inspected based on the average weld defect rate of the sub-images of vehicles manufactured in the factory within a first preset time period; for each vehicle element type, generating a change curve of the target weld defect rate of the vehicle element type within a second preset time period, and visually displaying the change curve; wherein the second preset time period is longer than the first preset time period.

[0082] For example, the system can statistically analyze the weld defect rate of various vehicle components, such as the body, cab, and frame, produced daily in a factory. Then, it can generate a weekly curve showing the change in the weld defect rate for each component over a week, and display this curve in a graphical user interface. This allows relevant personnel to analyze and repair the causes of weld defects in different components based on the changes in weld defect rates, thus helping to improve weld quality.

[0083] For example, the proportion of different types of weld defects among all weld defects in vehicles produced daily can be statistically analyzed, such as the proportion of porosity defects and crack defects, and variation curves for different types of defects can be generated. This can assist relevant personnel in conducting targeted analysis of the causes of different types of defects.

[0084] For example, the weld defect detection method in this disclosure may further include: statistically analyzing the defect level distribution of welds in the vehicle to be inspected; and generating weld quality inspection results for the vehicle to be inspected based on the defect level distribution.

[0085] For example, for a vehicle under inspection, the defect level distribution of the welds can be statistically analyzed, and a comprehensive weld quality inspection report can be generated based on this distribution. If any irreparable defect exists, the weld quality inspection result for the vehicle is deemed unqualified, and the report will then display the specific details of each defect level. If no irreparable defects exist, the weld quality inspection result is deemed qualified, and the report will also display all reparable defects, allowing relevant personnel to perform targeted repairs based on the report.

[0086] In one exemplary embodiment, the weld defect frame, weld defect type, weld defect size, weld defect level, and other relevant information can also be marked on the two-dimensional image corresponding to the vehicle to be inspected, and the weld defect area can be magnified for display.

[0087] In another exemplary implementation, the test results can be bound to the vehicle identification and stored in the database. For example, the test report, the two-dimensional image of the vehicle to be tested, and the vehicle identification of the vehicle to be tested can be associated and stored in the data to facilitate subsequent quality traceability.

[0088] For example, Figure 5 A schematic diagram of a weld defect detection system according to an exemplary embodiment of this disclosure is shown. (Reference) Figure 5When there is a need for vehicle weld defect detection, the automatic control system in the industrial computer 54 can send instructions to the work platform 52. The work platform 52 drives the industrial camera and LiDAR 51 to acquire images of the vehicle to be inspected, thereby obtaining an image of the vehicle. Then, the industrial camera transmits the image of the vehicle to be inspected to the industrial computer 54 through the local area network established by the switch 53. The industrial computer 54 can perform preprocessing on the image of the vehicle to be inspected, such as denoising and cropping the real weld image. Then, the preprocessed image is input into the weld defect detection model. The weld defect detection model detects and identifies defects in the real weld, obtaining the weld type, weld type, and weld defect type. The weld defect detection and analysis module in the industrial computer 54 can classify weld defects into defect levels, generate a visualized weld defect quality report based on the defect level classification results, and store and manage the weld defect quality report.

[0089] Furthermore, it should be noted that the above figures are merely illustrative representations of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0090] Furthermore, exemplary embodiments of this disclosure also provide a vehicle weld defect detection device. (See reference...) Figure 6 As shown, the vehicle weld defect detection device 600 includes the following program modules: a defect detection module 610, configured to acquire an image of the vehicle to be inspected, input the image of the vehicle to be inspected into a pre-trained weld defect detection model, and obtain the weld type, weld type, and weld defect type corresponding to the weld in the image of the vehicle to be inspected based on the output of the weld defect detection model; and a defect level determination module 620, configured to determine the defect level of the weld based on the weld type, weld type, and weld defect type corresponding to the weld, according to the defect size of the weld defect and a pre-trained weld defect level classification model.

[0091] In one exemplary embodiment, acquiring the vehicle image to be detected includes: acquiring a two-dimensional image and a three-dimensional point cloud image of the vehicle to be detected; based on this, inputting the vehicle image to be detected into a pre-trained weld defect detection model, and obtaining the weld type, weld type, and weld defect type of the weld in the vehicle to be detected according to the output of the weld defect detection model includes: inputting the two-dimensional image into a pre-trained weld recognition model, obtaining candidate welds in the two-dimensional image according to the output of the two-dimensional image; converting the two-dimensional coordinates of the candidate welds in the two-dimensional image into three-dimensional spatial coordinates through a pre-calibrated coordinate mapping model; determining the point cloud data corresponding to the candidate welds from the three-dimensional point cloud image based on the three-dimensional spatial coordinates; determining the actual welds from the candidate welds based on the point cloud data; and obtaining the weld type, weld type, and weld defect type of the weld in the vehicle to be detected based on the output of the pre-trained weld defect detection model according to the position of the two-dimensional image and the actual weld in the two-dimensional image.

[0092] In one exemplary embodiment, determining the actual weld from the candidate welds based on the point cloud data corresponding to the candidate welds includes: calculating the normal vector of a point in the point cloud data of the candidate weld; determining the angle between the normal vectors of adjacent points and counting a first number of determined angles; determining angles less than a threshold as target angles and calculating a first ratio between a second number of target angles and the first number; determining a suspected weld from the candidate welds based on the first ratio; fitting a curve of angle variation for the suspected weld and determining the actual weld from the suspected weld based on the curve of angle variation.

[0093] In one exemplary embodiment, the step of obtaining the weld type, weld type, and weld defect type corresponding to the weld in the vehicle to be inspected based on the output of a pre-trained weld defect detection model according to the position of the actual weld in the two-dimensional image and the actual weld in the two-dimensional image includes: cropping the actual weld image from the two-dimensional image according to the position of the actual weld in the two-dimensional image of the vehicle to be inspected; enlarging the actual weld image to obtain a target actual weld image corresponding to the actual weld image; inputting the target actual weld image into the pre-trained weld defect detection model, and obtaining the weld type, weld type, and weld defect type corresponding to the weld in the vehicle to be inspected based on the output of the weld defect detection model.

[0094] In one exemplary embodiment, determining the defect level of the weld based on the weld type, weld type, and weld defect type corresponding to the weld, and according to the defect size of the weld defect and a pre-trained weld defect level classification model, includes: calculating the defect size corresponding to the weld; inputting the weld type, weld type, weld defect type, and defect size corresponding to the weld into the pre-trained weld defect level classification model, and obtaining a first defect level of the weld based on the output of the weld defect level classification model; determining a second defect level corresponding to the defect size of the weld based on a preset mapping relationship between the weld defect size threshold and the defect level; if the first defect level and the second defect level are the same, determining the first defect level or the second defect level as the defect level of the weld; if the first defect level and the second defect level are different, obtaining the degree of difference between the defect size and the weld defect size threshold of the second defect level, and the output confidence of the weld defect level classification model, determining a target defect level from the first defect level and the second defect level based on the degree of difference and the output confidence, and determining the target defect level as the defect level of the weld.

[0095] In one exemplary embodiment, the vehicle to be inspected includes vehicles manufactured in a factory. The device further includes a change curve generation module, which can be configured to: divide the vehicle image to be inspected into regions according to vehicle element types to obtain sub-images to be inspected corresponding to each vehicle element type, wherein the vehicle element types include one or more of a cargo box, a cab, and a frame; calculate the weld defect rate in each sub-image to be inspected in each vehicle image to be inspected; for each sub-image to be inspected, obtain the target weld defect rate of the vehicle element type indicated by the sub-image to be inspected based on the average weld defect rate of the sub-images of vehicles to be inspected manufactured in the factory within a first preset time period; generate a change curve of the target weld defect rate of the vehicle element type within a second preset time period for each vehicle element type, and visualize the change curve; wherein the second preset time period is longer than the first preset time period.

[0096] In one exemplary embodiment, the method further includes a report generation module, which can be configured to: statistically analyze the defect level distribution of welds in the vehicle to be inspected; and generate weld quality inspection results for the vehicle to be inspected based on the defect level distribution.

[0097] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0098] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0099] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0100] An exemplary embodiment of this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the above-described vehicle weld defect detection method.

[0101] In one implementation, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0102] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0103] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, C++, and Python. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0104] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert the signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, the processor of the electronic device to execute) the method steps of various exemplary embodiments of this disclosure, such as the vehicle weld defect detection method described above.

[0105] Exemplary embodiments of this disclosure also provide an electronic device. The electronic device may include a processor and a memory. The memory stores executable instructions for the processor, such as a computer program. The processor executes the executable instructions to perform the method steps of various exemplary embodiments of this disclosure. Furthermore, the electronic device may also include a display for displaying a graphical user interface.

[0106] The following is for reference. Figure 7 The electronic device is illustrated by way of a general-purpose computing device. It should be understood that... Figure 7 The electronic device 700 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0107] like Figure 7 As shown, the electronic device 700 may include: a processor 710, a memory 720, a bus 730, an I / O (input / output) interface 740, a network adapter 750, and a display 760.

[0108] The memory 720 may include volatile memory, such as RAM 721 and cache unit 722, and may also include non-volatile memory, such as ROM 723. The memory 720 may also include one or more program modules 724, including but not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. For example, program module 724 may include the modules described above.

[0109] The processor 710 may include one or more processing units, such as an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-Network Processing Unit).

[0110] The processor 710 can be used to execute executable instructions stored in the memory 720, such as the vehicle weld defect detection method described above.

[0111] Bus 730 is used to connect different components of electronic device 700 and may include a data bus, an address bus and a control bus.

[0112] Electronic device 700 can communicate with one or more external devices 800 (such as keyboard, mouse, external controller, etc.) through I / O interface 740.

[0113] Electronic device 700 can communicate with one or more networks via network adapter 750. For example, network adapter 750 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication. Network adapter 750 can communicate with other modules of electronic device 700 via bus 730.

[0114] The electronic device 700 can display a graphical user interface via a display 760, such as displaying the weld quality inspection results of the vehicle to be inspected.

[0115] although Figure 7As not shown in the diagram, other hardware and / or software modules may also be configured in the electronic device 700, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0116] Those skilled in the art will understand that various aspects of this disclosure can be implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be referred to as "circuit", "module" or "system" respectively.

[0117] It should be understood that this disclosure is not limited to the specific methods, steps, or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. Those skilled in the art will readily conceive of other embodiments based on the specific implementations provided in this disclosure. Therefore, the specific implementations provided in this disclosure are merely exemplary, and the scope and spirit of this disclosure are indicated by the claims, and should cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary technical means in the art not disclosed in this disclosure.

Claims

1. A vehicle weld defect detection method, characterized by, The method comprises the steps of: collecting a to-be-detected vehicle image corresponding to a to-be-detected vehicle, inputting the to-be-detected vehicle image into a pre-trained welding seam defect detection model, and obtaining a welding seam category, a welding seam type and a welding seam defect type corresponding to a welding seam in the to-be-detected vehicle according to an output of the welding seam defect detection model; based on the welding seam category, the welding seam type and the welding seam defect type corresponding to the welding seam, determining a defect grade of the welding seam according to a defect size of the welding seam defect and a pre-trained welding seam defect grade division model.

2. The method of claim 1, wherein, The method comprises the steps of: collecting a two-dimensional image and a three-dimensional point cloud image corresponding to the to-be-detected vehicle; The method comprises the steps of: inputting the two-dimensional image into a pre-trained welding seam identification model, obtaining a candidate welding seam in the two-dimensional image according to an output of the two-dimensional image; converting two-dimensional coordinates of the candidate welding seam in the two-dimensional image into three-dimensional space coordinates through a pre-calibrated coordinate mapping model; determining point cloud data corresponding to the candidate welding seam from the three-dimensional point cloud image based on the three-dimensional space coordinates; determining a real welding seam from the candidate welding seam according to the point cloud data corresponding to the candidate welding seam; obtaining the welding seam category, the welding seam type and the welding seam defect type corresponding to the welding seam in the to-be-detected vehicle according to the two-dimensional image and a position of the real welding seam in the two-dimensional image based on an output of the pre-trained welding seam defect detection model.

3. The method of claim 2, wherein, The method comprises the steps of: calculating a normal vector of a point in the point cloud data of the candidate welding seam; determining an included angle of the normal vectors of adjacent points and counting a first number of the determined included angles; determining a target included angle that is smaller than a threshold value, and calculating a first ratio between a second number of the target included angles and the first number; determining a suspected welding seam from the candidate welding seam according to the first ratio; fitting a change curve of the included angles for the suspected welding seam, and determining the real welding seam from the suspected welding seam according to the change curve of the included angles.

4. The method of claim 2, wherein, The method comprises the steps of: cropping a real welding seam image from the two-dimensional image according to a position of the real welding seam in the two-dimensional image of the to-be-detected vehicle; enlarging the real welding seam image to obtain a target real welding seam image corresponding to the real welding seam image; inputting the target real welding seam image into the pre-trained welding seam defect detection model, and obtaining the welding seam category, the welding seam type and the welding seam defect type corresponding to the welding seam in the to-be-detected vehicle based on an output of the welding seam defect detection model.

5. The method of claim 1, wherein, The method comprises the following steps: calculating the defect size corresponding to the weld defect in the weld; inputting the weld type, the weld defect type and the defect size corresponding to the weld into the pre-trained weld defect level classification model, and obtaining the first defect level of the weld according to the output of the weld defect level classification model; determining the second defect level corresponding to the defect size of the weld according to the preset mapping relationship between the weld defect size threshold and the defect level; in the case that the first defect level and the second defect level are the same, the first defect level or the second defect level is determined as the defect level of the weld; in the case that the first defect level and the second defect level are not the same, the difference degree between the defect size and the weld defect size threshold of the second defect level is obtained, and the output confidence of the weld defect level classification model is obtained, and the target defect level is determined from the first defect level and the second defect level according to the difference degree and the output confidence, and the target defect level is determined as the defect level of the weld.

6. The method of claim 1, wherein, The vehicle to be detected includes a vehicle to be detected produced in a factory, and the method further comprises: dividing the vehicle to be detected image into regions according to the vehicle element type, to obtain a sub-detection image corresponding to each vehicle element type, wherein the vehicle element type includes one or more of a vehicle cabin, a vehicle head and a vehicle frame; statistically calculating the weld defect rate in each sub-detection image in each vehicle to be detected image; for each sub-detection image, obtaining a target weld defect rate of the vehicle element type indicated by the sub-detection image according to the average value of the weld defect rate of the sub-detection image of the vehicle to be detected produced by the factory within a first preset time period; for each vehicle element type, generating a change curve diagram of the target weld defect rate of the vehicle element type within a second preset time period, and visually displaying the change curve diagram; wherein the second preset time period is greater than the first preset time period.

7. The method of claim 1, wherein, The method further comprises: statistically calculating the defect level distribution of the weld in the vehicle to be detected; generating a weld quality detection result of the vehicle to be detected according to the defect level distribution.

8. A vehicle weld defect detection apparatus characterized by, It comprises: a defect detection module configured to collect a vehicle to be detected image, input the vehicle to be detected image into a pre-trained weld defect detection model, and obtain the weld type, the weld defect type and the weld defect type corresponding to the weld in the vehicle to be detected image according to the output of the weld defect detection model; a defect level determination module configured to determine the defect level of the weld based on the weld type, the weld defect type and the weld defect type corresponding to the weld according to the defect size of the weld defect and a pre-trained weld defect level classification model.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the method of any one of claims 1 to 7.

10. An electronic device, comprising: It comprises: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to carry out the method of any one of claims 1 to 7.

Images