Detection method, device, equipment and system for preventing misloading and neglected loading in ZP8 final inspection and storage medium

Abstract

The invention provides a ZP8 final inspection mistake and neglected loading prevention detection method, device, equipment and system and a storage medium. The method comprises the following steps: acquiring a KNR number of a to-be-detected vehicle; determining the type and configuration of the to-be-detected vehicle according to the KNR number of the to-be-detected vehicle; determining a currently used trained detection model corresponding to the vehicle type and the configuration; acquiring real shot images of a plurality of parts of the to-be-detected vehicle; inputting the actually shot image into a currently used trained detection model to obtain a detected PR number of the part; according to the KNR number of the to-be-detected vehicle, vehicle configuration information of the to-be-detected vehicle is obtained from a production control system, and the vehicle configuration information comprises configuration part PR number information; consistency comparison is carried out on the detected PR number of the part and the PR number information of the configured part, and a detection result is generated; and generating alarm information under the condition that the detected PR number of the part is inconsistent with the PR number information of the configured part. According to the invention, the cost of automatic detection is reduced, the detection rate of wrong and neglected loading defects is improved, and the workload of inspectors is reduced.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of vehicle detection technology, and more specifically, to a ZP8 final inspection error-proof missing installation detection method, device, equipment, system and storage medium. Background Art

[0002] Automobile manufacturers have been facing pain points due to the mixed production of multiple models and the variety of vehicle configurations. Missing parts and misconfiguration have always been a pain point for all OEMs. It is difficult to identify misconfiguration and missing parts during manual final inspection. The final inspection lacks efficient and accurate defect identification methods, resulting in defect outflow.

[0003] At present, traditional machine vision inspection technology can be used to identify image features and realize inspection functions. However, the traditional machine vision method has the problem of manually designed feature extraction algorithm, which has problems such as poor robustness and poor adaptability. Summary of the invention

[0004] In order to solve the above-mentioned problems in the prior art, in the first aspect, an embodiment of the present invention provides a ZP8 final inspection error-prevention and missing assembly detection method, the method comprising: obtaining the KNR number of a vehicle to be inspected; determining the model and configuration of the vehicle to be inspected according to the KNR number of the vehicle to be inspected; determining the currently used trained detection model corresponding to the model and configuration of the vehicle to be inspected; obtaining real-shot images of multiple parts of the vehicle to be inspected; inputting the real-shot images into the currently used trained detection model to obtain the PR number of the detected part; according to the KNR number of the vehicle to be inspected, obtaining the vehicle configuration information of the vehicle to be inspected from the production control system, the vehicle configuration information including the PR number information of the configured part; performing a consistency comparison between the detected part PR number and the configured part PR number information to generate a detection result; if the detection result is that the detected part PR number is inconsistent with the configured part PR number information, generating an alarm message.

[0005] In some embodiments, the trained detection model is obtained by the following steps: for each model among multiple model types and each configuration among multiple configurations of each model, the initial detection model is trained using sample images of vehicles of each model and each configuration as a training data set to obtain a trained detection model for each model and each configuration.

[0006] In some embodiments, obtaining the real-shot images of the multiple parts of the vehicle to be inspected includes: performing a predetermined number of image acquisitions on the multiple parts of the vehicle to be inspected at a predefined shooting interval to obtain a predetermined number of groups of real-shot images. Comparing the detected part PR number with the configured part PR number information for consistency, and generating the test result includes: comparing the detected part PR number in each of the predetermined number of groups of real-shot images with the configured part PR number information for consistency, to obtain a predetermined number of single test results; and taking the comprehensive value of the predetermined number of single test results as the test result.

[0007] In some embodiments, the method further includes: inputting the real-shot image into the currently used trained detection model to obtain an image with a marked detection frame.

[0008] In some implementations, the method further includes: storing the configuration part PR number information and the detection result in a database.

[0009] In some embodiments, the method further includes: displaying one or more of the following: the KNR number of the vehicle, the model and configuration of the vehicle to be inspected, the real-shot image, the detected part PR number, the configured part PR number information, the detection result, and the alarm information.

[0010] In the second aspect, an embodiment of the present invention proposes a ZP8 final inspection error-proof missing assembly detection device, the device comprising: a KNR number acquisition module, configured to acquire the KNR number of a vehicle to be inspected; a vehicle model configuration determination module, configured to determine the vehicle model and configuration of the vehicle to be inspected according to the KNR number of the vehicle to be inspected; a detection model determination module, configured to determine the currently used trained detection model corresponding to the vehicle model and configuration of the vehicle to be inspected; a real-shot image acquisition module, configured to acquire real-shot images of multiple parts of the vehicle to be inspected; a detection module, configured to input the real-shot image into the currently used trained detection model to obtain the detected part PR number; a configuration information acquisition module, configured to acquire the vehicle configuration information of the vehicle to be inspected from the production control system according to the KNR number of the vehicle to be inspected, the vehicle configuration information including the configuration part PR number information; a comparison module, configured to compare the detected part PR number with the configuration part PR number information for consistency, and generate a detection result; an alarm module, configured to generate an alarm message when the detection result is that the detected part PR number is inconsistent with the configuration part PR number information.

[0011] In the third aspect, an embodiment of the present invention proposes a ZP8 final inspection and error-proof missing installation detection device, the device comprising a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the ZP8 final inspection and error-proof missing installation detection method described in any of the above embodiments is implemented.

[0012] In the fourth aspect, the embodiment of the present invention proposes a ZP8 final inspection error prevention missing installation detection system, the system includes a barcode scanner, a camera, an alarm device and the ZP8 final inspection error prevention missing installation detection device described above. The barcode scanner is used to scan the fender label of the vehicle to be inspected, obtain the KNR number of the vehicle to be inspected, and transmit the KNR number of the vehicle to be inspected to the ZP8 final inspection error prevention missing installation detection device, the camera is used to collect real-shot images of multiple parts of the vehicle to be inspected, and transmit the real-shot images to the ZP8 final inspection error prevention missing installation detection device, and the alarm device is used to perform an alarm action when receiving the alarm information sent by the ZP8 final inspection error prevention missing installation detection device.

[0013] In a fifth aspect, an embodiment of the present invention proposes a storage medium storing computer-readable instructions, which, when executed by a processor, executes the ZP8 final inspection error-proof missing installation detection method described in any of the above embodiments.

[0014] The implementation method of the present invention proposes an automatic detection solution for the problem of wrong or missing assembly of the whole vehicle. By combining and building multiple network cameras, the whole vehicle can be monitored in real time. Based on the deep learning algorithm framework, multiple types of parts are sampled and trained to realize automatic recognition of part models. An error-proofing program is developed to automatically compare the part model detection results with the consistency of the whole vehicle configuration. When wrong or missing assembly occurs, the inspector is prompted to deal with the defect through the on-site display, alarm lights and buzzers.

[0015] The embodiments of the present invention realize a whole vehicle anti-error and missed assembly visual inspection solution based on deep learning algorithms and network cameras, and utilize deep learning-based adaptive image feature detection technology to realize the inspection function, thus solving the problem of automatic inspection in the industry in a very low-cost manner.

[0016] The present invention proposes a detection scheme for machine vision identification of wrong or missing assembly defects of a whole vehicle developed based on a deep learning algorithm model. The detection rate of wrong or missing assembly defects is improved through machine vision detection, 100% interception of defects is achieved, and the workload of inspectors is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0018] Figure 1 A flow chart of a ZP8 final inspection error-proof missing installation detection method according to an embodiment of the present invention is shown;

[0019] Figure 2 A flowchart showing an example procedure of a ZP8 final inspection error-proofing and missing installation detection method according to an embodiment of the present invention;

[0020] Figure 3 A deployment diagram of a ZP8 final inspection error-proofing and missing installation detection system according to an embodiment of the present invention is shown.

[0021] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0022] The principle and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way.

[0023] In one aspect, an embodiment of the present invention provides a ZP8 final inspection error prevention missing detection method, referring to Figure 1 , which shows a flow chart of a ZP8 final inspection error-proof missing installation detection method according to an embodiment of the present invention. The method includes steps S101-S108.

[0024] In step S101, the KNR number (body control number) of the vehicle to be inspected is obtained. For example, the vehicle KNR number can be read by scanning the fender label of the vehicle to be inspected with a barcode scanner. For another example, the barcode scanner information can be obtained through the TCP / IP protocol to obtain the vehicle KNR number (body control number).

[0025] In actual application scenarios, after the barcode scanner scans the fender label to read the vehicle KNR number, it triggers the detection start signal and starts running the detection program.

[0026] In step S102, the model and configuration of the vehicle to be inspected are determined according to the KNR number of the vehicle to be inspected.

[0027] In step S103, the currently used trained detection model corresponding to the vehicle model and configuration of the vehicle to be detected is determined. The detection program can automatically switch the corresponding detection model according to different vehicle models.

[0028] As an embodiment of the present invention, a trained detection model is obtained by the following steps: for each model among multiple model types and each configuration among multiple configurations of each model type, sample images of vehicles of each model type and each configuration are used as training data sets to train the initial detection model to obtain a trained detection model for each model type and each configuration.

[0029] For example, the initial detection model can adopt a variety of machine vision deep learning algorithms, among which mainstream algorithms include Fast-CNN, SSD, and yolo algorithms. Preferably, the yolo algorithm can be used, which can meet the scene requirements of the present invention in terms of timeliness and accuracy. As an example only, a neural network model based on the Yolov8s framework can be made, using the pytorch deep learning framework and socket communication. The deep learning algorithm based on yolo effectively balances the requirements of detection accuracy and efficiency and meets the needs of actual scenarios.

[0030] As an example, each time the detection model is trained, 50-100 images of each model and each configuration are used as the training data set. Taking the Magotan tail label as an example, there are three configurations of "280TSI", "330TSI" and "380TSI", with a total of 284 images. The number of images in the test data set is 29, which is 10% of the training data set.

[0031] In step S104, real images of multiple parts of the vehicle to be inspected are obtained.

[0032] As an example only, four network cameras may be used for video capture to capture images of the left exterior, left interior, right exterior, and right interior of the vehicle respectively.

[0033] In step S105, the real shot image is input into the currently used trained detection model to obtain the detected part PR number (part code). As an embodiment of the present invention, an image with a marked detection frame can also be obtained.

[0034] In step S106 , vehicle configuration information of the vehicle to be inspected is acquired from a production control system (eg, a FIS system) according to the KNR number of the vehicle to be inspected, wherein the vehicle configuration information includes configuration part PR number information.

[0035] In step S107, the detected part PR number is compared with the configured part PR number information for consistency to generate a detection result.

[0036] In step S108, if the detected part PR number is inconsistent with the configured part PR number information, an alarm message is generated. For example, the alarm message can be transmitted to an on-site alarm device, such as a display, a speaker, a buzzer, an alarm light-emitting device, etc., to prompt the staff at the production site.

[0037] As an embodiment of the present invention, since there are influencing factors such as personnel and vehicle driving in the ZP8 final inspection chain, multiple image acquisitions can be performed. Step S104 may include: performing a predetermined number of image acquisitions on multiple parts of the vehicle to be inspected at a predefined shooting interval (for example, 3 seconds), for example, performing 50 times for each vehicle, and obtaining a predetermined number of groups of real-shot images. In this embodiment, step S107 may include: performing consistency comparisons on the part PR numbers detected in each of the predetermined number of groups of real-shot images with the configured part PR number information, respectively, to obtain a predetermined number of single detection results; and taking the comprehensive value of the predetermined number of single detection results as the detection result. Taking the execution of 50 image acquisitions as an example, 50 inspections and consistency comparisons are performed on each vehicle, and the comprehensive value of 50 rounds of inspections is calculated as the final inspection result.

[0038] As an embodiment of the present invention, the method may further include: storing the configuration part PR number information and the test results in a database to facilitate subsequent traceability query.

[0039] As an embodiment of the present invention, the method may also include displaying one or more of the following: the vehicle's KNR number, the model and configuration of the vehicle to be inspected, the real-shot image, the detected part PR number, the configured part PR number information, the inspection results, and the alarm information.

[0040] refer to Figure 2 , which shows a flowchart of an example procedure of the ZP8 final inspection error prevention missing detection method according to an embodiment of the present invention. In the figure, "Y" means "yes" and "N" means "no".

[0041] When the barcode scanner recognizes the vehicle KNR number, a trigger signal is generated and the detection program starts.

[0042] Load the detection model. As an example, the detection model can be obtained in the following ways: collect vehicle material pictures, use Labelimg software to annotate the material pictures with part features (parts involved include, for example, steering wheels, shift knobs, dashboard trims, wheels, privacy glass, tail labels, etc.), a total of 25,446 vehicle model training materials, and divide the training data set and test data set into a 9:1 ratio. Based on the training data set, the Yolov8s model is used for deep learning training, and the test data set is used to verify the model accuracy, and the detection model is loaded into the detection program.

[0043] The detection program takes a photo every 3 seconds and inputs the photo into the detection program. The target recognition threshold is set to 0.85, that is, when the target confidence is greater than 85%, it is judged as a valid target and the detection result is output. A total of 50 photos are taken for each vehicle. The final detection result of the vehicle is obtained and displayed on the QT front-end interface after combining the 50 detection results.

[0044] The vehicle configuration information is obtained from the FIS system and compared with the test results. If the configuration information of a part is inconsistent with the test results, it is judged as "unqualified" and an alarm is sounded by the speaker at the production site.

[0045] Write vehicle configuration information and test results into the MySQL database for easy subsequent query and tracing.

[0046] Optionally, Tensortflow can be used to improve the model lightweight, improve the detection speed of the detection model while ensuring the model detection accuracy, and quickly detect the entire vehicle.

[0047] On the other hand, the embodiment of the present invention proposes a ZP8 final inspection error prevention missing detection device, which includes the following modules:

[0048] A KNR number acquisition module configured to acquire the KNR number of the vehicle to be inspected;

[0049] A vehicle model configuration determination module, configured to determine the vehicle model and configuration of the vehicle to be inspected according to the KNR number of the vehicle to be inspected;

[0050] a detection model determination module configured to determine a currently used trained detection model corresponding to the model and configuration of the vehicle to be inspected;

[0051] A real-shot image acquisition module configured to acquire real-shot images of multiple parts of the vehicle to be inspected;

[0052] A detection module, configured to input the real shot image into the currently used trained detection model to obtain the PR number of the detected part;

[0053] A configuration information acquisition module is configured to acquire vehicle configuration information of the vehicle to be inspected from the production control system according to the KNR number of the vehicle to be inspected, wherein the vehicle configuration information includes PR number information of configuration parts;

[0054] A comparison module is configured to compare the detected part PR number with the configured part PR number information for consistency and generate a detection result;

[0055] The alarm module is configured to generate an alarm message when the detection result shows that the detected part PR number is inconsistent with the configured part PR number information.

[0056] It should be noted that the functions implemented by each module in the ZP8 final inspection and anti-error missing installation detection device proposed in the embodiment of the present invention correspond one-to-one to the various steps of the ZP8 final inspection and anti-error missing installation detection method described above. For its specific implementation methods, examples and beneficial effects, please refer to the above description of the method.

[0057] On the other hand, an embodiment of the present invention proposes a ZP8 final inspection and error-proof missing installation detection device, the device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the ZP8 final inspection and error-proof missing installation detection method described in any of the above implementation schemes is implemented.

[0058] In another aspect, the embodiments of the present invention provide a ZP8 final inspection error prevention and missing detection system. Figure 3 , which shows a schematic diagram of the deployment of the ZP8 final inspection error prevention missing detection system according to an embodiment of the present invention. The system includes a barcode scanner, a camera, an alarm device, and the ZP8 final inspection error prevention missing detection device described above.

[0059] The barcode scanner is used to scan the fender label of the vehicle to be inspected, obtain the KNR number of the vehicle to be inspected, and transmit the KNR number of the vehicle to be inspected to the ZP8 final inspection error-proofing and missing installation detection equipment.

[0060] The camera is used to collect real images of multiple parts of the vehicle to be inspected, and transmit the real images to the ZP8 final inspection and error-proofing and missing inspection equipment.

[0061] The alarm device is used to execute the alarm action when receiving the alarm information sent by the ZP8 final inspection error prevention and missing installation detection equipment.

[0062] As an example, four network cameras can be installed in the ZP8 final inspection chain to capture images of the left exterior, left interior, right exterior, and right interior of the vehicle respectively. A barcode scanner can be installed to scan the vehicle fender label to obtain the vehicle KNR number. A PC host equipped with an NVIDIA independent graphics card can be installed. The camera, barcode scanner, and PC host can be connected with an Internet cable, and the detection program can be deployed in the PC host.

[0063] In this example, a common network camera is used for hardware construction, which greatly reduces the cost compared with the method of building industrial cameras and light sources in the industry. However, the present invention is not limited to network cameras.

[0064] For example, by building a parallel structure with multiple detection processes, multiple cameras can be used for real-time monitoring and the detection results can be combined and compared.

[0065] On the other hand, an embodiment of the present invention provides a storage medium storing computer-readable instructions, which, when executed by a processor, executes the ZP8 final inspection error-proof missing installation detection method described in any of the above embodiments.

[0066] The implementation method of the present invention proposes an automatic detection solution for the problem of wrong or missing assembly of the whole vehicle. By combining and building multiple network cameras, the whole vehicle can be monitored in real time. Based on the deep learning algorithm framework, multiple types of parts are sampled and trained to realize automatic recognition of part models. An error-proofing program is developed to automatically compare the part model detection results with the consistency of the whole vehicle configuration. When wrong or missing assembly occurs, the inspector is prompted to deal with the defect through the on-site display, alarm lights and buzzers.

[0067] The embodiments of the present invention realize a whole vehicle anti-error and missed assembly visual inspection solution based on deep learning algorithms and network cameras, and utilize deep learning-based adaptive image feature detection technology to realize the inspection function, thus solving the problem of automatic inspection in the industry in a very low-cost manner.

[0068] The present invention is a detection solution for machine vision recognition of wrong or missing assembly defects of a whole vehicle developed based on a deep learning algorithm model. The detection rate of wrong or missing assembly defects is improved through machine vision detection, 100% defect interception is achieved, and the workload of inspectors is reduced.

[0069] For the purpose of illustration, the foregoing description of the embodiments of the present invention has been given, which is not exhaustive nor intended to limit the present invention to the disclosed exact form. It will be appreciated by those skilled in the art that various changes can be made without departing from the scope of the present invention, and that the elements therein can be replaced with equivalents. In addition, without departing from the basic scope of the present invention, many modifications can be made so that specific situations or materials are adapted to the teachings of the present invention. Therefore, the present invention is not intended to be limited to the specific embodiments disclosed as the best mode for realizing the present invention, and the present invention will include all embodiments falling within the scope of the appended claims.

Claims

1. A ZP8 final inspection error and missing detection method, characterized in that: The method comprises: Obtain the KNR number of the vehicle to be inspected; Determine the model and configuration of the vehicle to be inspected according to the KNR number of the vehicle to be inspected; Determining a currently used trained detection model corresponding to the model and configuration of the vehicle to be inspected; Acquire real-shot images of multiple parts of the vehicle to be inspected; Input the real shot image into the currently used trained detection model to obtain the PR number of the detected part; According to the KNR number of the vehicle to be inspected, the vehicle configuration information of the vehicle to be inspected is obtained from the production control system, wherein the vehicle configuration information includes the PR number information of the configuration parts; Comparing the detected part PR number with the configured part PR number information for consistency, and generating a detection result; When the detection result shows that the detected part PR number is inconsistent with the configured part PR number information, an alarm message is generated.

2. The method according to claim 1, characterized in that The trained detection model is obtained by the following steps: For each model among the multiple models and each configuration of the multiple configurations of each model, the initial detection model is trained using sample images of vehicles of each model and each configuration as a training data set to obtain a trained detection model for each model and each configuration.

3. The method according to claim 1, characterized in that Acquiring real-shot images of multiple parts of the vehicle to be inspected includes: performing a predetermined number of image acquisitions on the multiple parts of the vehicle to be inspected at a predefined shooting interval to obtain a predetermined number of groups of real-shot images, and, Comparing the detected part PR number with the configured part PR number information for consistency, and generating a test result includes: Comparing the PR number of the part detected in each of the predetermined number of groups of real-shot images with the PR number information of the configured part for consistency, to obtain a predetermined number of single detection results; The comprehensive value of the predetermined number of single detection results is used as the detection result.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The real-shot image is input into the currently used trained detection model to obtain an image with a marked detection frame.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: The configuration part PR number information and the detection result are stored in a database.

6. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Display one or more of the following: the KNR number of the vehicle, the model and configuration of the vehicle to be inspected, the real-shot image, the PR number of the detected part, the PR number information of the configured part, the inspection result, and the alarm information.

7. A ZP8 final inspection error and missing detection device, characterized in that: The device comprises: A KNR number acquisition module configured to acquire the KNR number of the vehicle to be inspected; A vehicle model configuration determination module, configured to determine the vehicle model and configuration of the vehicle to be inspected according to the KNR number of the vehicle to be inspected; a detection model determination module configured to determine a currently used trained detection model corresponding to the model and configuration of the vehicle to be inspected; A real-shot image acquisition module, configured to acquire real-shot images of multiple parts of the vehicle to be inspected; A detection module, configured to input the real-shot image into the currently used trained detection model to obtain the PR number of the detected part; A configuration information acquisition module, configured to acquire vehicle configuration information of the vehicle to be inspected from a production control system according to the KNR number of the vehicle to be inspected, wherein the vehicle configuration information includes configuration part PR number information; A comparison module, configured to compare the detected part PR number with the configured part PR number information for consistency, and generate a detection result; The alarm module is configured to generate an alarm message when the detection result shows that the detected part PR number is inconsistent with the configured part PR number information.

8. A ZP8 final inspection error and missing detection device, characterized in that: The device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the ZP8 final inspection error-proof missing installation detection method described in any one of claims 1 to 6 is implemented.

9. A ZP8 final inspection error and missing detection system, characterized in that: The system includes a barcode scanner, a camera, an alarm device, and the ZP8 final inspection error prevention and missing detection device according to claim 8, wherein: The barcode scanner is used to scan the fender label of the vehicle to be inspected, obtain the KNR number of the vehicle to be inspected, and transmit the KNR number of the vehicle to be inspected to the ZP8 final inspection error prevention and missing detection equipment. The camera is used to collect real images of multiple parts of the vehicle to be inspected, and transmit the real images to the ZP8 final inspection error prevention and missing detection equipment, The alarm device is used to execute an alarm action when receiving the alarm information sent by the ZP8 final inspection error prevention and missing installation detection equipment.

10. A storage medium, characterized in that: Computer-readable instructions are stored, and when the instructions are executed by the processor, the ZP8 final inspection error-proofing and missing installation detection method according to any one of claims 1-6 is executed.

Images