Computer Vision-Based Vehicle Frame Quality Inspection Method and System

Abstract

This invention discloses a computer vision-based method and system for inspecting the quality of vehicle frames, comprising: setting two positioning points at preset positions on a standard vehicle frame and storing a front view image of the standard vehicle frame as a reference image; setting two positioning points at preset positions on the vehicle frame to be inspected, and acquiring a front view image of the vehicle frame to be inspected through an image acquisition device as an image to be inspected; performing semantic segmentation on the reference image and the image to be inspected to generate a reference binary image and a binary image to be inspected; performing an exhaustive combination transformation on the vehicle frame entities in the binary image to be inspected until the two positioning points of the binary image to be inspected coincide with the two positioning points of the reference binary image; comparing the transformed binary image to be inspected with the reference binary image to generate a frame error index, and determining whether the quality of the vehicle frame to be inspected is qualified. This invention realizes automated computer vision inspection of the production quality of the entire vehicle frame.

Description

Technical Field

[0001] This invention relates to the field of computer vision inspection, and more specifically to a method and system for inspecting the quality of vehicle frames based on computer vision. Background Technology

[0002] In the production process of electric tricycles, information technology and automation are often used to improve production efficiency and product qualification rate. In this process, computer vision technology is a common technical means to inspect the production quality of the whole vehicle frame, so as to verify whether the actual size, shape, position tolerance and other aspects of the components meet the design specifications.

[0003] However, in actual production lines, the placement of the frames to be inspected is usually not completely consistent. This makes it difficult for ordinary inspection methods to perform uniform automated inspection of the production quality of the entire vehicle frame. In addition, the structure of the entire vehicle frame is complex, and ordinary image segmentation models are unable to accurately segment the edges between the frame entity and the background in the image of the entire vehicle frame, resulting in insufficient accuracy of the inspection method. Summary of the Invention

[0004] The purpose of this invention is to provide a computer vision-based method and system for inspecting vehicle frame quality. This invention aims to address the problem that in actual production lines, the placement of vehicle frames to be inspected is often not completely consistent. This makes it difficult for ordinary inspection methods to perform uniform automated inspection of the production quality of the entire vehicle frame. Furthermore, the structure of the entire vehicle frame is complex, and ordinary image segmentation models are unable to accurately segment the edges between the frame entity and the background in the image of the entire vehicle frame, resulting in insufficient accuracy of the inspection method.

[0005] In view of the above problems, this application provides a computer vision-based method and system for inspecting the quality of vehicle frames.

[0006] The first aspect disclosed in this application provides a computer vision-based method for inspecting the quality of a vehicle frame, the method comprising:

[0007] Two positioning points are set at preset positions on the standard frame, and the front view image of the standard frame is stored as a reference image;

[0008] Two positioning points are set at preset positions on the vehicle frame to be inspected, and the front view image of the vehicle frame to be inspected is obtained through an image acquisition device as the image to be inspected;

[0009] An HRNet image segmentation model is constructed to perform semantic segmentation on the reference image and the image to be detected, segmenting the standard frame entity and the frame entity to be detected in the reference image and the image to be detected, and generating the reference binary image and the image to be detected binary image.

[0010] The vehicle frame entity to be detected in the binary image to be detected is subjected to an exhaustive combination of scaling, rotation and translation transformations until two positioning points on the vehicle frame entity to be detected in the binary image to be detected coincide with two positioning points on the standard vehicle frame entity in the reference binary image, thereby generating the transformed binary image to be detected.

[0011] The transformed binary image to be detected is XORed with the reference binary image to generate a deviation binary image. The ratio of non-zero pixels in the deviation binary image and the reference binary image is calculated to generate the frame error index.

[0012] The quality of the frame under test is determined by whether the frame error index exceeds a preset threshold.

[0013] Preferably, the construction of the HRNet image segmentation model, which performs semantic segmentation on the reference image and the image to be detected, specifically includes:

[0014] The reference image and the image to be detected are input into the neck network for downsampling, and the initial reference feature map and the initial image to be detected are generated sequentially.

[0015] The initial reference feature map and the initial feature map to be detected are respectively input into the first-stage network for downsampling, and the first reference feature map and the first feature map to be detected are generated sequentially.

[0016] The first reference feature map and the first feature map to be detected are respectively input into the second-stage network for downsampling, and the second reference feature map and the second feature map to be detected are generated sequentially.

[0017] The second reference feature map and the second feature map to be detected are respectively input into the third-stage network for downsampling, and the third reference feature map and the third feature map to be detected are generated sequentially.

[0018] The third reference feature map and the third feature map to be detected are respectively input into the fourth stage network for downsampling, and the fourth reference feature map and the fourth feature map to be detected are generated in sequence.

[0019] The fourth reference feature map and the fourth detection feature map are fused separately, and the class probability of each pixel in the reference image and the detection image is generated by 1×1 convolution calculation and activation function calculation, thus generating semantic segmentation results.

[0020] Preferably, the resolution of the initial reference feature map and the initial feature map to be detected is half that of the reference image and the image to be detected; the resolution of the first reference feature map and the first feature map to be detected is half that of the reference image and the image to be detected; the second reference feature map and the second feature map to be detected each contain two branches, with resolutions of half and one-quarter that of the reference image and the image to be detected, respectively; the third reference feature map and the third feature map to be detected each contain three branches, with resolutions of half, one-quarter, and one-eighth that of the reference image and the image to be detected, respectively; and the fourth reference feature map and the fourth feature map to be detected each contain four branches, with resolutions of half, one-quarter, one-eighth, and one-tenth that of the reference image and the image to be detected, respectively.

[0021] Preferably, the step of performing an exhaustive combination of scaling, rotation, and translation transformations on the vehicle frame entity in the binary image to be detected until two positioning points on the vehicle frame entity in the binary image to be detected coincide with two positioning points on the standard vehicle frame entity in the reference binary image specifically includes:

[0022] Using the binary image to be detected and the reference binary image as references, establish a Cartesian coordinate system respectively;

[0023] Obtain the coordinates of two positioning points on the standard frame entity in the reference binary image, and obtain the coordinates of two positioning points on the frame entity to be detected in the binary image to be detected;

[0024] A homogeneous transformation matrix is ​​used to exhaustively combine scaling, rotation, and translation transformations on the vehicle frame entity to be detected in the binary image to be detected. The coordinate positions of the two positioning points on the vehicle frame entity to be detected in the binary image to be detected are also transformed accordingly until the transformed coordinate positions are the same as the coordinate positions of the two positioning points on the standard vehicle frame entity in the reference binary image.

[0025] The second aspect of this application discloses a computer vision-based frame quality inspection system, the system being used in the aforementioned computer vision-based frame quality inspection method, the system comprising:

[0026] The reference image module is used to set two positioning points at preset positions on the standard frame and store the front view image of the standard frame as a reference image.

[0027] The image module to be inspected is used to set two positioning points at a preset position on the frame to be inspected, and to acquire a front view image of the frame to be inspected through an image acquisition device as the image to be inspected;

[0028] The image segmentation module is used to construct the HRNet image segmentation model, perform semantic segmentation on the reference image and the image to be detected, segment out the standard frame entity and the frame entity to be detected in the reference image and the image to be detected, and generate the reference binary image and the binary image to be detected.

[0029] The combined transformation module is used to perform exhaustive scaling, rotation and translation combined transformations on the vehicle frame entity to be detected in the binary image to be detected until two positioning points on the vehicle frame entity to be detected in the binary image to be detected coincide with two positioning points on the standard vehicle frame entity in the reference binary image, thereby generating the transformed binary image to be detected.

[0030] The logic operation module is used to perform an XOR operation on the transformed binary image to be detected and the reference binary image to generate a deviation binary image, calculate the ratio of non-zero pixels in the deviation binary image and the reference binary image, and generate the frame error index.

[0031] The determination module is used to determine whether the quality of the frame to be tested is qualified based on whether the frame error index exceeds a preset threshold.

[0032] The third aspect disclosed in this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described computer vision-based frame quality inspection method.

[0033] The fourth aspect disclosed in this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described computer vision-based frame quality inspection method.

[0034] The fifth aspect disclosed in this application provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described computer vision-based frame quality inspection method.

[0035] The beneficial effects of this invention are as follows:

[0036] (1) By using the method of setting positioning points and exhaustive combination transformation, the quality inspection of the frame to be inspected at any position in the production line can be realized, and the production quality of the whole vehicle frame can be automatically inspected by computer vision.

[0037] (2) By performing multiple feature extractions through the multi-stage downsampling operation of the HRNet model and preserving the high-resolution details of the image through the multi-branch structure of the feature map, accurate semantic segmentation of the complex vehicle frame is achieved, which improves the accuracy of the detection method. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of the overall process for a computer vision-based vehicle frame quality inspection method.

[0040] Figure 2 This is a diagram showing the overall structure of a computer vision-based vehicle frame quality inspection system. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Example 1:

[0043] like Figure 1 As shown in the figure, this application provides a computer vision-based method for inspecting the quality of a vehicle frame, the method comprising:

[0044] Step 1: Set two positioning points at the preset positions on the standard frame and store the front view image of the standard frame as a reference image.

[0045] Step 2: Set two positioning points at preset positions on the frame to be inspected, and acquire the front view image of the frame to be inspected using an image acquisition device as the image to be inspected.

[0046] Step 3: Construct the HRNet image segmentation model. The HRNet image segmentation model includes a neck network, a first-stage network, a second-stage network, a third-stage network, and a fourth-stage network.

[0047] The reference image and the image to be detected are input into the neck network for downsampling, and the initial reference feature map and the initial image to be detected are generated sequentially.

[0048] The initial reference feature map and the initial feature map to be detected are respectively input into the first-stage network for downsampling, and the first reference feature map and the first feature map to be detected are generated sequentially.

[0049] The first reference feature map and the first feature map to be detected are respectively input into the second-stage network for downsampling, and the second reference feature map and the second feature map to be detected are generated sequentially.

[0050] The second reference feature map and the second feature map to be detected are respectively input into the third-stage network for downsampling, and the third reference feature map and the third feature map to be detected are generated sequentially.

[0051] The third reference feature map and the third feature map to be detected are respectively input into the fourth stage network for downsampling, and the fourth reference feature map and the fourth feature map to be detected are generated in sequence.

[0052] The fourth reference feature map and the fourth detection feature map are fused separately, and the class probability of each pixel in the reference image and the detection image is generated by 1×1 convolution calculation and activation function. Semantic segmentation is performed on the reference image and the detection image to segment the standard frame entity and the detection frame entity in the reference image and the detection image, and the reference binary image and the detection binary image are generated.

[0053] The resolutions of the initial reference feature map and the initial feature map to be detected are half that of the reference image and the image to be detected. The resolutions of the first reference feature map and the first feature map to be detected are half that of the reference image and the image to be detected. The second reference feature map and the second feature map to be detected each contain two branches, with resolutions of half and one-quarter that of the reference image and the image to be detected, respectively. The third reference feature map and the third feature map to be detected each contain three branches, with resolutions of half, one-quarter, and one-eighth that of the reference image and the image to be detected, respectively. The fourth reference feature map and the fourth feature map to be detected each contain four branches, with resolutions of half, one-quarter, one-eighth, and one-tenth that of the reference image and the image to be detected, respectively.

[0054] Step 4: Perform exhaustive scaling, rotation and translation transformations on the vehicle frame entity to be detected in the binary image to be detected until two positioning points on the vehicle frame entity to be detected in the binary image to be detected coincide with two positioning points on the standard vehicle frame entity in the reference binary image, thereby generating the transformed binary image to be detected.

[0055] The specific steps include:

[0056] Step 4.1: Establish a Cartesian coordinate system based on the binary image to be detected and the reference binary image respectively;

[0057] Step 4.2: Obtain the coordinates of two positioning points on the standard frame entity in the reference binary image, and obtain the coordinates of two positioning points on the frame entity to be detected in the binary image to be detected.

[0058] Step 4.3: Use a homogeneous transformation matrix to perform an exhaustive combination of scaling, rotation and translation transformations on the vehicle frame entity to be detected in the binary image to be detected. The coordinate positions of the two positioning points on the vehicle frame entity to be detected in the binary image to be detected are also transformed accordingly until the transformed coordinate positions are the same as the coordinate positions of the two positioning points on the standard vehicle frame entity in the reference binary image.

[0059] Step 5: Perform an XOR operation between the transformed binary image to be detected and the reference binary image to generate a deviation binary image. Calculate the ratio of non-zero pixels in the deviation binary image and the reference binary image to generate the frame error index.

[0060] Step 6: Determine whether the quality of the frame to be tested is qualified based on whether the frame error index exceeds the preset threshold.

[0061] In addition to the steps mentioned above, the HRNet image segmentation model also needs to be trained. The training steps include:

[0062] Images of the electric tricycle frame are acquired, all images are normalized and input dimensions are standardized, and then the image data is augmented by rotation and flipping to alleviate the problem of insufficient image data. Finally, manual annotation is performed, and the annotations are converted into One-Hot encoding or binary mask form to build a dataset, which is divided into training set, validation set and test set in an 8:1:1 ratio.

[0063] Dice Loss was used to optimize overlapping regions, and weighted cross-entropy loss was used to alleviate the frame-background pixel imbalance problem. The AdamW optimizer (learning rate 3e-4) was used to train the HRNet image segmentation model using the training set, and the accuracy was verified using the validation set. The accuracy was represented by IoU and Recall.

[0064] Early stopping is employed; if the accuracy on the validation set does not increase for 10 consecutive epochs, model training is stopped, and then the HRNet image segmentation model is tested on the test set.

[0065] In summary, the computer vision-based vehicle frame quality inspection method provided in this application has the following technical effects:

[0066] (1) By using the method of setting positioning points and exhaustive combination transformation, the quality inspection of the frame to be inspected at any position in the production line can be realized, and the production quality of the whole vehicle frame can be automatically inspected by computer vision.

[0067] (2) By performing multiple feature extractions through the multi-stage downsampling operation of the HRNet model and preserving the high-resolution details of the image through the multi-branch structure of the feature map, accurate semantic segmentation of the complex vehicle frame is achieved, which improves the accuracy of the detection method.

[0068] Example 2:

[0069] Based on the same inventive concept as the computer vision-based frame quality inspection method in Embodiment 1, such as Figure 2 As shown, this application provides a computer vision-based vehicle frame quality inspection system, the system comprising:

[0070] The reference image module is used to set two positioning points at preset positions on the standard frame and store the front view image of the standard frame as a reference image.

[0071] The image module to be inspected is used to set two positioning points at a preset position on the frame to be inspected, and to acquire a front view image of the frame to be inspected through an image acquisition device as the image to be inspected;

[0072] The image segmentation module is used to construct the HRNet image segmentation model, perform semantic segmentation on the reference image and the image to be detected, segment out the standard frame entity and the frame entity to be detected in the reference image and the image to be detected, and generate the reference binary image and the binary image to be detected.

[0073] The combined transformation module is used to perform exhaustive scaling, rotation and translation combined transformations on the vehicle frame entity to be detected in the binary image to be detected until two positioning points on the vehicle frame entity to be detected in the binary image to be detected coincide with two positioning points on the standard vehicle frame entity in the reference binary image, thereby generating the transformed binary image to be detected.

[0074] The logic operation module is used to perform an XOR operation on the transformed binary image to be detected and the reference binary image to generate a deviation binary image, calculate the ratio of non-zero pixels in the deviation binary image and the reference binary image, and generate the frame error index.

[0075] The determination module is used to determine whether the quality of the frame to be tested is qualified based on whether the frame error index exceeds a preset threshold.

[0076] Through the foregoing detailed description of the computer vision-based frame quality inspection method, those skilled in the art can clearly understand the computer vision-based frame quality inspection system in this embodiment. Since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and relevant parts can be found in the method section.

[0077] Example 3:

[0078] In Embodiment 3, a computer device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the above-described computer vision-based vehicle frame quality inspection method.

[0079] Example 4:

[0080] In Embodiment 4, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the above-described computer vision-based frame quality inspection method.

[0081] Example 5:

[0082] In Embodiment 5, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-described computer vision-based frame quality inspection method.

[0083] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A computer vision-based method for inspecting the quality of vehicle frames, characterized in that, The method includes: Two positioning points are set at preset positions on the standard frame, and the front view image of the standard frame is stored as a reference image; Two positioning points are set at preset positions on the vehicle frame to be inspected, and the front view image of the vehicle frame to be inspected is obtained through an image acquisition device as the image to be inspected; An HRNet image segmentation model is constructed to perform semantic segmentation on the reference image and the image to be detected, segmenting the standard frame entity and the frame entity to be detected in the reference image and the image to be detected, and generating the reference binary image and the image to be detected binary image. The vehicle frame entity to be detected in the binary image to be detected is subjected to an exhaustive combination of scaling, rotation and translation transformations until two positioning points on the vehicle frame entity to be detected in the binary image to be detected coincide with two positioning points on the standard vehicle frame entity in the reference binary image, thereby generating the transformed binary image to be detected. The transformed binary image to be detected is XORed with the reference binary image to generate a deviation binary image. The ratio of non-zero pixels in the deviation binary image and the reference binary image is calculated to generate the frame error index. The quality of the frame under test is determined by whether the frame error index exceeds a preset threshold.

2. The computer vision-based frame quality inspection method as described in claim 1, characterized in that, The construction of the HRNet image segmentation model, which performs semantic segmentation on the reference image and the image to be detected, specifically includes: The reference image and the image to be detected are input into the neck network for downsampling, and the initial reference feature map and the initial image to be detected are generated sequentially. The initial reference feature map and the initial feature map to be detected are respectively input into the first-stage network for downsampling, and the first reference feature map and the first feature map to be detected are generated sequentially. The first reference feature map and the first feature map to be detected are respectively input into the second-stage network for downsampling, and the second reference feature map and the second feature map to be detected are generated sequentially. The second reference feature map and the second feature map to be detected are respectively input into the third-stage network for downsampling, and the third reference feature map and the third feature map to be detected are generated sequentially. The third reference feature map and the third feature map to be detected are respectively input into the fourth stage network for downsampling, and the fourth reference feature map and the fourth feature map to be detected are generated in sequence. The fourth reference feature map and the fourth detection feature map are fused separately, and the class probability of each pixel in the reference image and the detection image is generated by 1×1 convolution calculation and activation function calculation, thus generating semantic segmentation results.

3. The computer vision-based vehicle frame quality inspection method as described in claim 2, characterized in that, The resolution of the initial reference feature map and the initial feature map to be detected is half that of the reference image and the image to be detected. The resolution of the first reference feature map and the first feature map to be detected is half that of the reference image and the image to be detected. The second reference feature map and the second feature map to be detected each contain two branches, with resolutions of half and one-quarter that of the reference image and the image to be detected, respectively. The third reference feature map and the third feature map to be detected each contain three branches, with resolutions of half, one-quarter, and one-eighth that of the reference image and the image to be detected, respectively. The fourth reference feature map and the fourth feature map to be detected each contain four branches, with resolutions of half, one-quarter, one-eighth, and one-tenth that of the reference image and the image to be detected, respectively.

4. The computer vision-based vehicle frame quality inspection method as described in claim 1, characterized in that, The process of exhaustively scaling, rotating, and translating the vehicle frame entity in the binary image to be detected until two positioning points on the vehicle frame entity in the binary image to be detected coincide with two positioning points on the standard vehicle frame entity in the reference binary image specifically includes: Using the binary image to be detected and the reference binary image as references, establish a Cartesian coordinate system respectively; Obtain the coordinates of two positioning points on the standard frame entity in the reference binary image, and obtain the coordinates of two positioning points on the frame entity to be detected in the binary image to be detected; A homogeneous transformation matrix is ​​used to exhaustively combine scaling, rotation, and translation transformations on the vehicle frame entity to be detected in the binary image to be detected. The coordinate positions of the two positioning points on the vehicle frame entity to be detected in the binary image to be detected are also transformed accordingly until the transformed coordinate positions are the same as the coordinate positions of the two positioning points on the standard vehicle frame entity in the reference binary image.

5. A computer vision-based vehicle frame quality inspection system, the system comprising: The reference image module is used to set two positioning points at preset positions on the standard frame and store the front view image of the standard frame as a reference image. The image module to be inspected is used to set two positioning points at a preset position on the frame to be inspected, and to acquire a front view image of the frame to be inspected through an image acquisition device as the image to be inspected; The image segmentation module is used to construct the HRNet image segmentation model, perform semantic segmentation on the reference image and the image to be detected, segment out the standard frame entity and the frame entity to be detected in the reference image and the image to be detected, and generate the reference binary image and the binary image to be detected. The combined transformation module is used to perform exhaustive scaling, rotation and translation combined transformations on the vehicle frame entity to be detected in the binary image to be detected until two positioning points on the vehicle frame entity to be detected in the binary image to be detected coincide with two positioning points on the standard vehicle frame entity in the reference binary image, thereby generating the transformed binary image to be detected. The logic operation module is used to perform an XOR operation on the transformed binary image to be detected and the reference binary image to generate a deviation binary image, calculate the ratio of non-zero pixels in the deviation binary image and the reference binary image, and generate the frame error index. The determination module is used to determine whether the quality of the frame to be tested is qualified based on whether the frame error index exceeds a preset threshold.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the computer vision-based frame quality inspection method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the computer vision-based frame quality inspection method according to any one of claims 1 to 4.

8. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps of the computer vision-based frame quality inspection method according to any one of claims 1 to 4.

Images