Vehicle final assembly detection method, device and system and electronic equipment

Abstract

The invention provides a vehicle final assembly detection method, device and system and electronic equipment. According to the invention, the image acquisition point location, the part area needing to be detected and the corresponding part detection method are configured for the target vehicle in advance, and at least one part area matched with the image acquisition point location can be identified from the vehicle image under the condition that the vehicle image of the target vehicle acquired at the image acquisition point location is obtained. Furthermore, for each part area, the part area is detected according to the part detection method matched with the part corresponding to the part area, the detection result of each part area is obtained, the vehicle final assembly detection result is generated, automatic detection of the vehicle final assembly is achieved, and the detection efficiency and the detection accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to vehicle assembly inspection methods, devices, systems and electronic equipment. Background Technology

[0002] Vehicle final assembly is the process of assembling vehicle parts into a complete vehicle according to predetermined design, process requirements, and order configurations. The core objective of conformity testing is to ensure that each assembled vehicle strictly conforms to its design specifications and the specific configuration required by the user. Currently, conformity testing in automotive final assembly workshops still relies primarily on manual verification, that is, manually observing whether the vehicle matches the actual required configuration. This method is inefficient, has a high false positive rate, and carries a significant risk of missed detections. Summary of the Invention

[0003] In view of this, this application provides vehicle assembly testing methods, apparatus, systems and electronic equipment to improve testing efficiency and reduce false detection rate.

[0004] The technical solution provided in this application is as follows: According to an embodiment of the first aspect of this application, a vehicle final assembly inspection method is provided, the method being applied to electronic devices, the method comprising: Obtain vehicle images of the target vehicle captured by the image acquisition device at the image acquisition point; Identify at least one part region from the vehicle image that matches the image acquisition point; different part regions correspond to different parts on the target vehicle; For each part area, the part area is inspected according to the part inspection method matched to the part corresponding to that part area; wherein, the inspection results of each part area are used to generate the final assembly inspection results of the target vehicle.

[0005] Optionally, the vehicle image includes: An image acquisition device fixed at a designated image acquisition point, capturing a vehicle image of the target vehicle when the target vehicle, located on a conveying device, is transported to a position matching the designated image acquisition point; and / or, An image acquisition device mounted on a robot's robotic arm acquires vehicle images of the target vehicle when the robotic arm moves along a specified trajectory to a specified location, wherein the specified location includes at least the interior of the target vehicle.

[0006] Optionally, the part region is detected according to the part detection method matched with the part corresponding to the part region, including: If the part corresponding to the part region is a first designated part, and the first designated part is a part in the target vehicle that needs to be detected to exist, then the region to which the part belongs is detected based on the trained contour detection model. If the part corresponding to the part region is the second designated part, and the second designated part is the part in the target vehicle that needs to be detected as to whether it is marked, then the part region is detected based on the trained mark detection model; If the part corresponding to the part region is a third designated part, and the third designated part is a part in the target vehicle whose color needs to be detected, then the part region is detected based on the trained color recognition model. If the part corresponding to the part region is the fourth designated part, and the fourth designated part is the part in the target vehicle whose shape needs to be detected, then the part region is detected based on the trained shape detection model.

[0007] Optionally, the electronic device is connected to an external aggregation station via HTTP communication; after detecting each part area according to the part detection method corresponding to the part in that part area, the method further includes: The test results are sent to the aggregation industrial control computer in JSON string format via HTTP protocol. The aggregation industrial control computer then organizes the test results of each part area to generate the final assembly test results of the target vehicle.

[0008] Optionally, the image acquisition point corresponding to the target vehicle, the part region matching the image acquisition point, and the part detection method matching the part corresponding to the part region are determined based on pre-stored configuration information; the configuration information is generated through the following steps: When a detection parameter configuration operation for the target vehicle model is detected, a first configuration interface is output. The first configuration interface is used to configure image acquisition points for the target vehicle corresponding to the target vehicle model. Upon detecting a configuration operation for the target image acquisition point in the first configuration interface, a second configuration interface is output. The second configuration interface is used to configure the part region to be detected and the corresponding part detection type for the target image acquisition point; different part detection types use different part detection methods. Based on the configuration operations received in the first configuration interface and the second configuration interface, configuration information matching the target vehicle corresponding to the target model is generated and stored.

[0009] Optionally, the first configuration interface includes a point list area and an overview area; the point list area records at least one image acquisition point configured for the target vehicle, and each image acquisition point is associated with device identification information, which indicates whether the image acquisition device at the image acquisition point is a fixed image acquisition device or an image acquisition device mounted on the robot's robotic arm; when any image acquisition point in the point list area is selected, the overview area displays the image currently acquired by the image acquisition device corresponding to that image acquisition point; The second configuration interface includes a parts list area and a parameter setting area; the parts list area includes the parts to be detected at the target image acquisition point, the part area to which each part belongs in the image acquired at the target image acquisition point, and the part detection type of each part; the parameter setting area includes parameter configuration information of at least one image acquisition device configured at the target image acquisition point, and the parameter configuration information includes at least one of exposure parameters, gain parameters, Gamma parameters, and process parameters.

[0010] According to an embodiment of the second aspect of this application, a vehicle final assembly inspection system is provided, comprising: At least one electronic device for performing the method as described in the first aspect; The aggregation station is connected to each electronic device via HTTP communication. The aggregation station is used to collect the part inspection results sent by each electronic device for the target vehicle, and generate the final assembly inspection results of the target vehicle.

[0011] According to an embodiment of a third aspect of this application, a vehicle final assembly testing apparatus is provided, which is applied to electronic equipment, and the apparatus includes: The acquisition unit is used to acquire vehicle images of the target vehicle captured by the image acquisition device at the image acquisition point. The identification unit is used to identify at least one part region from the vehicle image that matches the image acquisition point; different part regions correspond to different parts on the target vehicle; The detection unit is used to detect each part area according to the part detection method matched to the part corresponding to that part area; wherein, the detection results of each part area are used to generate the final assembly detection results of the target vehicle.

[0012] Optionally, the vehicle image includes: An image acquisition device fixed at a designated image acquisition point, capturing a vehicle image of the target vehicle when the target vehicle, located on a conveying device, is transported to a position matching the designated image acquisition point; and / or, An image acquisition device mounted on a robot's robotic arm, which acquires vehicle images of the target vehicle when the robotic arm moves along a specified trajectory to a specified location of the target vehicle, wherein the specified location includes at least the interior of the target vehicle; And / or, the detection unit is specifically used for: If the part corresponding to the part region is a first designated part, and the first designated part is a part in the target vehicle that needs to be detected to exist, then the region to which the part belongs is detected based on the trained contour detection model. If the part corresponding to the part region is the second designated part, and the second designated part is the part in the target vehicle that needs to be detected as to whether it is marked, then the part region is detected based on the trained mark detection model; If the part corresponding to the part region is a third designated part, and the third designated part is a part in the target vehicle whose color needs to be detected, then the part region is detected based on the trained color recognition model. If the part corresponding to the part region is the fourth designated part, and the fourth designated part is the part in the target vehicle whose shape needs to be detected, then the part region is detected based on the trained shape detection model. And / or, the electronic device is connected to an external aggregation station via HTTP communication; after detecting each part area according to the part detection method corresponding to the part in that part area, the detection unit is further configured to: The test results are sent to the aggregation industrial control computer in JSON string format via HTTP protocol, so that the aggregation industrial control computer can organize the test results of each part area and generate the final assembly test results of the target vehicle. And / or, the image acquisition point corresponding to the target vehicle, the part region matching the image acquisition point, and the part detection method matching the part corresponding to the part region are determined based on pre-stored configuration information; the configuration information is generated through the following steps: When a detection parameter configuration operation for the target vehicle model is detected, a first configuration interface is output. The first configuration interface is used to configure image acquisition points for the target vehicle corresponding to the target vehicle model. Upon detecting a configuration operation for the target image acquisition point in the first configuration interface, a second configuration interface is output. The second configuration interface is used to configure the part region to be detected and the corresponding part detection type for the target image acquisition point; different part detection types use different part detection methods. Based on the configuration operations received in the first configuration interface and the second configuration interface, configuration information matching the target vehicle corresponding to the target vehicle model is generated and stored; And / or, the first configuration interface includes a point list area and an overview area; the point list area records at least one image acquisition point configured for the target vehicle, and each image acquisition point is associated with device identification information, which indicates whether the image acquisition device at the image acquisition point is a fixed image acquisition device or an image acquisition device mounted on the robot's robotic arm; when any image acquisition point in the point list area is selected, the overview area displays the image currently acquired by the image acquisition device corresponding to that image acquisition point; The second configuration interface includes a parts list area and a parameter setting area; the parts list area includes the parts to be detected at the target image acquisition point, the part area to which each part belongs in the image acquired at the target image acquisition point, and the part detection type of each part; the parameter setting area includes parameter configuration information of at least one image acquisition device configured at the target image acquisition point, and the parameter configuration information includes at least one of exposure parameters, gain parameters, Gamma parameters, and process parameters.

[0013] According to an embodiment of the fourth aspect of this application, an electronic device is provided, comprising: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; the processor is configured to execute the machine-executable instructions to implement the method described in the first aspect.

[0014] As can be seen from the above technical solution, this application pre-configures image acquisition points, part areas to be detected, and corresponding part detection methods for the target vehicle. When a vehicle image of the target vehicle is acquired at the image acquisition point, at least one part area matching the image acquisition point can be identified from the vehicle image. Furthermore, for each part area, the part area is detected according to the part detection method matching the part corresponding to that part area, and the detection results of each part area are obtained to generate the vehicle assembly inspection results. This realizes the automatic inspection of vehicle assembly and improves the inspection efficiency and accuracy. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the principles of this application.

[0016] Figure 1 This is a flowchart of a vehicle assembly and testing method provided in an embodiment of this application. Figure 2This is a schematic diagram of a camera mounting scheme provided in an embodiment of this application; Figure 3 This is a schematic diagram of the vehicle parameter configuration tool provided in an embodiment of this application; Figure 4 This is a network topology diagram of a vehicle assembly and testing system provided in an embodiment of this application. Figure 5 A communication interaction timing diagram provided for an embodiment of this application; Figure 6A A visual inspection flowchart provided for embodiments of this application; Figure 6B A specific visual inspection flowchart is provided for an embodiment of this application; Figure 7 A flowchart of data loading and visual inspection provided in this application embodiment; Figure 8 This is a schematic diagram of a vehicle assembly and testing system provided in an embodiment of this application; Figure 9 A structural diagram of a vehicle assembly and testing device provided in this application embodiment; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0018] Vehicle final assembly refers to the process of assembling vehicle parts into a complete vehicle according to predetermined design, process requirements, and order configurations. Consistency testing of vehicle final assembly involves systematically inspecting each assembled vehicle to ensure that all assembled parts strictly conform to their design specifications and the specific configuration required by the user.

[0019] For example, specific consistency checks include, but are not limited to, checking whether parts exist, whether parts are correctly installed, and whether the color and shape of parts are accurate.

[0020] Currently, the conformity inspection of automobile assembly in related technologies still mainly relies on manual verification. This involves quality inspectors checking each vehicle on the assembly line against the configuration list to observe whether the vehicle matches the actual required configuration. This manual inspection method is highly subjective and easily affected by factors such as the fatigue and experience of the quality inspectors, resulting in a high risk of missed or false positives, making it difficult to guarantee consistent inspection quality.

[0021] Based on this, this application proposes a vehicle assembly inspection method to improve inspection efficiency and reduce false detection rate.

[0022] Please refer to Figure 1 , Figure 1 A flowchart illustrating the vehicle assembly and testing method provided in this application embodiment.

[0023] In this embodiment, the method can be applied to electronic devices, such as intuitive processing industrial control computers, programmable logic controllers, edge computing devices, etc., and this application does not impose any limitations on them.

[0024] like Figure 1 As shown, the method may include the following steps: Step 101: Obtain the vehicle image of the target vehicle captured by the image acquisition device at the image acquisition point.

[0025] In this embodiment, for different vehicle models, at least one image acquisition point can be pre-set according to the parts that need to be inspected in that vehicle model, to ensure that the images acquired at the image acquisition point include the parts that need to be inspected. The process of pre-setting the image acquisition point will be described in detail below and will not be repeated here.

[0026] Specific methods for obtaining vehicle images of the target vehicle captured by the image acquisition device at the image acquisition point may include: Based on the vehicle model, determine the corresponding image acquisition points for the target vehicle, and acquire images of the target vehicle according to the corresponding image acquisition points to obtain the vehicle image.

[0027] The vehicle image of the target vehicle may include: An image acquisition device fixed at a designated image acquisition point, capturing a vehicle image of the target vehicle when the target vehicle, located on a conveying device, is transported to a position matching the designated image acquisition point; and / or, An image acquisition device mounted on a robot's robotic arm acquires vehicle images of the target vehicle when the robotic arm moves along a specified trajectory to a specified location, wherein the specified location includes at least the interior of the target vehicle.

[0028] In this embodiment, considering that when the target vehicle is being inspected during final assembly, the target vehicle is moved as a whole by a conveyor on the final assembly line, and the image acquisition device is controlled to acquire images of the target vehicle at the corresponding image acquisition points, when it is necessary to acquire images of the vehicle's interior, the mechanical structure of the fixed image acquisition device may interfere with the vehicle, leading to problems such as difficulty in acquiring images of the vehicle's interior and engine compartment, and low image quality. Therefore, this application proposes two image acquisition methods: acquiring images of the target vehicle through an image acquisition device fixed at a designated image acquisition point, and acquiring images of the target vehicle through an image acquisition device mounted on the robotic arm of a robot.

[0029] For details, please refer to Figure 2 , Figure 2 This is a schematic diagram of a camera mounting scheme provided in an embodiment of this application.

[0030] like Figure 2 As shown, this camera setup scheme employs a hybrid setup method that combines a fixed camera with a camera carried by a collaborative robotic arm.

[0031] Several industrial cameras (such as...) are fixedly installed above and on both sides of the vehicle inspection line. Figure 2 The fixed camera group 1, fixed camera group 2 and fixed camera group 3 shown in the figure (each fixed camera group can deploy one or more cameras) are precisely calibrated to quickly acquire images of the area outside the vehicle when the vehicle travels to a preset specific location.

[0032] In addition, at key locations on the testing line (such as...) Figure 2 At the crossbeam in the middle, at least one collaborative robotic arm (such as...) is deployed. Figure 2 The device connects camera 1 to the crossbeam and camera 2 to the crossbeam. The camera is integrated on the end flange of the robotic arm, and a matching light source can also be deployed there. In this embodiment, this design allows the camera to move with the collaborative robotic arm, overcoming the physical limitations of fixed installation. When the vehicle travels to a preset specific position, the robotic arm moves into the vehicle's interior to collect images of the vehicle's interior.

[0033] Specifically, sensors (such as photoelectric sensors) installed on the detection line can be used to determine whether a target vehicle has entered the workstation. After confirming that the target vehicle has entered the workstation, the programmable logic controller (PLC) can read the encoder value synchronized with the conveyor in real time, thereby tracking the vehicle's position based on the encoder value.

[0034] When the target vehicle is moved to a preset specific position based on the change in encoder value, the camera at the fixed position is controlled to take a picture to acquire images of the exterior of the target vehicle. Additionally, a robotic arm is controlled to extend into the interior of the target vehicle in a specified posture to take pictures of designated areas inside the vehicle, thus solving the problem of image acquisition for internal vehicle inspection items (such as bolts, connectors, plugs, etc.).

[0035] This concludes the discussion on... Figure 2 The description.

[0036] After obtaining the vehicle image of the target vehicle, step 101 ends, and step 102 is executed next.

[0037] Step 102: Identify at least one part region from the vehicle image that matches the image acquisition point.

[0038] Different part areas correspond to different parts on the target vehicle.

[0039] In this embodiment, the vehicle images of the target vehicle acquired at each image acquisition point can be pre-calibrated, that is, the region of interest (the region to which the part to be detected belongs) is marked in the vehicle images of the target vehicle acquired at each image acquisition point. Then, when the vehicle images of the target vehicle are acquired at each image acquisition point in the future, the part region to which the part to be detected belongs can be determined according to the position of the pre-calibrated region of interest in the vehicle image.

[0040] This concludes the description of step 102. We will now proceed to step 103.

[0041] Step 103: For each part area, the part area is detected according to the part detection method matched with the part corresponding to that part area.

[0042] The inspection results of each part area are used to generate the final assembly inspection results of the target vehicle.

[0043] As one embodiment, the part region is detected according to the part detection method matching the part corresponding to the part region, including: If the part corresponding to the part region is a first designated part, and the first designated part is a part in the target vehicle that needs to be detected to exist, then the region to which the part belongs is detected based on the trained contour detection model. If the part corresponding to the part region is the second designated part, and the second designated part is the part in the target vehicle that needs to be detected as to whether it is marked, then the part region is detected based on the trained mark detection model; If the part corresponding to the part region is a third designated part, and the third designated part is a part in the target vehicle whose color needs to be detected, then the part region is detected based on the trained color recognition model. If the part corresponding to the part region is the fourth designated part, and the fourth designated part is the part in the target vehicle whose shape needs to be detected, then the part region is detected based on the trained shape detection model.

[0044] In this embodiment, based on the current visual inspection requirements of the final assembly process, the inspection types can be divided into the following categories: Presence / Absence Detection: This involves detecting whether a part exists, is incorrectly assembled, or is missing, to reflect whether there are any omissions or errors in the final assembly process. The part that needs to be detected, i.e. the part whose existence needs to be checked, is designated as the first specified part.

[0045] Color mark inspection: During the automotive assembly process, colored markers are used to mark parts such as bolts and connectors to confirm whether certain processes have been completed. The parts that need to be color-marked, i.e. the parts that need to be checked for marking, are referred to as the second designated parts.

[0046] Color Inspection: In automotive assembly, some interior and exterior parts may have the same shape but different colors, and there may be multiple color combinations. To confirm whether these parts have been installed with the wrong color, color inspection is necessary. Here, the parts that need color inspection, i.e., the parts whose colors need to be checked, are referred to as the third specified parts.

[0047] Shape detection: For some parts with complex shapes and compositions, greater susceptibility to imaging interference, or weak configuration number definition rules, shape detection can be used to detect their configuration numbers. The parts that need to be shape detected are referred to as the fourth specified parts.

[0048] In this embodiment, each part region is pre-configured with a matching part detection method. It should be noted that the image acquisition point corresponding to the target vehicle, the part region matching the image acquisition point, and the part detection method matching the part corresponding to the part region are determined based on pre-stored configuration information.

[0049] Specifically, the configuration information is generated through the following steps: When a detection parameter configuration operation for the target vehicle model is detected, a first configuration interface is output. The first configuration interface is used to configure image acquisition points for the target vehicle corresponding to the target vehicle model. Upon detecting a configuration operation for the target image acquisition point in the first configuration interface, a second configuration interface is output. The second configuration interface is used to configure the part region to be detected and the corresponding part detection type for the target image acquisition point; different part detection types use different part detection methods. Based on the configuration operations received in the first configuration interface and the second configuration interface, configuration information matching the target vehicle corresponding to the target model is generated and stored.

[0050] Before implementing the vehicle assembly inspection method proposed in this application, the information to be inspected for each vehicle model can be configured in advance. For example, the vehicle model configuration and other interactive buttons can be clicked on the interactive interface of the electronic device. When the electronic device determines that the detection parameter configuration operation for the target vehicle model has been detected, the first configuration interface is output on the interactive interface. The first configuration interface is used to configure the image acquisition points for the target vehicle model, that is, to configure the camera position for acquiring the vehicle model.

[0051] If a configuration operation is detected for the target image acquisition point in the first configuration interface, a second configuration interface is output to further configure the part area to be detected and the corresponding part detection type for the target image acquisition point.

[0052] Specifically, the first configuration interface includes a point list area and an overview area; the point list area records at least one image acquisition point configured for the target vehicle, and each image acquisition point is associated with device identification information, which indicates whether the image acquisition device at that point is a fixed image acquisition device or an image acquisition device mounted on the robot's robotic arm; when any image acquisition point in the point list area is selected, the overview area displays the image currently acquired by the image acquisition device corresponding to that image acquisition point; The second configuration interface includes a parts list area and a parameter setting area; the parts list area includes the parts to be detected at the target image acquisition point, the part area to which each part belongs in the image acquired at the target image acquisition point, and the part detection type of each part; the parameter setting area includes parameter configuration information of at least one image acquisition device configured at the target image acquisition point, and the parameter configuration information includes at least one of exposure parameters, gain parameters, Gamma parameters, and process parameters.

[0053] Please refer to Figure 3 , Figure 3 This is a schematic diagram of a vehicle parameter configuration tool provided in an embodiment of this application.

[0054] like Figure 3As shown, in this embodiment, an automotive configuration tool was developed in conjunction with the parts management method in the final assembly process. This tool stores information such as vehicle model, part number, inspection type, and inspection location. The automotive configuration tool can be deployed on electronic devices that perform vehicle final assembly inspection.

[0055] in, Figure 3 The point configuration interface on the left is the first configuration interface mentioned above. In this interface, users can add, delete, or modify the image acquisition points of the target vehicle currently being configured. Each image acquisition point can be set to image acquisition by a fixed camera or a camera on a robotic arm. Users can also view the image acquisition effect of a selected image acquisition point in the overview area.

[0056] In this embodiment, the image acquisition point can be added or deleted, or the image acquisition method corresponding to the point can be modified, by using the modification button corresponding to each image acquisition point.

[0057] Meanwhile, each image acquisition point can be further configured. Clicking the configuration button corresponding to any image acquisition point will take you to the parameter configuration interface for that image acquisition point, i.e., the second configuration interface mentioned above.

[0058] In the parameter configuration interface, you can specifically set the parameters of at least one camera deployed at the image acquisition point, such as exposure parameters, gain parameters, flow parameters, gamma parameters, etc. This application does not impose any restrictions on this.

[0059] The parameter configuration interface also includes a list of parts matched to the image acquisition point. The list of parts records the parts that need to be detected at the image acquisition point. For each part, the corresponding detection type is also recorded. Different detection types correspond to different part detection methods. For example, for color detection, a trained color detection model is used for detection. This application does not impose any restrictions on this.

[0060] In addition, for each part, the region of interest (ROI) in the vehicle image captured at the image acquisition point is also recorded. This is so that when the vehicle assembly inspection is carried out, after obtaining the vehicle image corresponding to the vehicle model captured at the image acquisition point, the part region corresponding to the part can be determined from the vehicle image based on the aforementioned ROI.

[0061] As an example, for each part, image features (such as the edge contour of the part) within the ROI area can be extracted and saved as a template file. This template file is then bound to the vehicle model, image acquisition points, and the part and stored locally on the electronic device.

[0062] This concludes the discussion on... Figure 3 The description.

[0063] This concludes the description of step 103.

[0064] After determining the inspection results of each part area according to steps 101 to 103, the inspection results of each part area can be summarized to obtain the final assembly inspection results of the target vehicle.

[0065] This concludes the discussion on... Figure 1 The description.

[0066] This application pre-configures image acquisition points, component regions to be detected, and corresponding component detection methods for the target vehicle. Upon obtaining a vehicle image of the target vehicle acquired at the image acquisition points, at least one component region matching the image acquisition point can be identified from the vehicle image. Furthermore, for each component region, the component region is detected according to the component detection method corresponding to that component region, obtaining the detection results for each component region to generate the vehicle assembly inspection results. This achieves automatic vehicle assembly inspection, improving inspection efficiency and accuracy.

[0067] As an example, in order to facilitate the summarization of the inspection results of each part inspection area, this application sets up an external summarization station machine.

[0068] Please refer to Figure 4 The network topology diagram of the vehicle assembly and testing system provided in this application embodiment.

[0069] like Figure 4 As shown, the MOMO system is a workshop management system, which can be one or more high-performance servers. The MOMO system stores the standard configuration list of each model. For example, the standard configuration of the low-end version of model A includes which parts, the color and shape of each part, and the standard configuration of the high-end version of model A includes which parts, the color and shape of each part, etc. This application does not impose any restrictions on this.

[0070] In this embodiment, the MOMO system only interacts with the newly added aggregation station. It sends the model of the current target vehicle (i.e., the vehicle to be inspected) to the aggregation station and receives the vehicle assembly results aggregated by the aggregation station after the inspection is completed.

[0071] Specifically, each electronic device connects to the external aggregation station via HTTP communication. After inspecting each part area according to the corresponding part inspection method, the electronic device sends the inspection results as JSON strings to the aggregation industrial control computer via HTTP. The aggregation industrial control computer then processes the inspection results for each part area to generate the final assembly inspection results for the target vehicle. The data aggregation industrial control computer performs data processing, recording, displaying, storing, and analyzing the data, ultimately uploading the final assembly inspection results to the MOMO system. It can highlight NG (Not Good Information) information and images, enabling real-time uploading of NG information and control of the production line through the MOMO system.

[0072] The aggregation station receives vehicle instructions from MOMO and distributes them to the corresponding downstream production line stations (e.g., [missing information]) via a system-level network switch. Figure 4 The system (including workstations at the interior, chassis, and external pipeline lines) receives test results reported by all workstations on the production line and integrates them to generate a complete and final assembly test report for the vehicle.

[0073] Each production line workstation is an electronic device that performs the vehicle final assembly inspection method described in this application. Different production line workstations (e.g., chassis, exterior, interior) can be configured for inspection, and the results are aggregated by a single consolidation workstation. In this embodiment, backup production line workstations can be configured for each workstation, for example... Figure 4 The backup production line station machine in the external pipeline is used to switch to the backup production line station machine for testing when the current production line station machine fails.

[0074] Among them, MOMO, the aggregation station machine, and the station machines of each production line are connected to the system-level network switch to provide a stable and high-speed network connection, ensuring that instructions and data can be transmitted reliably.

[0075] For each workstation on the production line, there is one or more industrial cameras ( Figure 4 The image acquisition device mentioned in this application includes interior line cameras, chassis line cameras, and external pipeline cameras, which are used to acquire vehicle images of the target vehicle (exterior of the vehicle or through a robotic arm) at various image acquisition points.

[0076] Each production line workstation is connected to its corresponding camera via a workstation-level network switch.

[0077] This concludes the discussion on... Figure 4 The description.

[0078] The following is through Figure 5 The above description Figure 4 The collaborative workflow of various architectures within it.

[0079] Please refer to Figure 5 , Figure 5 This is a timing diagram of communication interaction provided for an embodiment of this application.

[0080] like Figure 5 As shown, MOMO refers to the aforementioned workshop management system, and the inspection line refers to the framework software within the workstations of the aforementioned production line. This framework software has HTTP sending and receiving capabilities and can support various functions such as vehicle information reception, parts query, and result feedback.

[0081] Specifically, after a vehicle enters the workstation, MOMO first sends the vehicle model information to the aggregation workstation. Upon receiving the model information, the aggregation workstation forwards it to the framework software of the corresponding inspection line. The framework software, upon receiving the model information, sends back a pre-configured list of parts to be inspected for that model. This list is then forwarded to MOMO by the aggregation workstation. MOMO queries the standard parts list for the current vehicle (i.e., the standard parts configuration of the current vehicle, such as the presence, color, and shape of parts) and sends it back to the framework software via the aggregation workstation. The framework software then performs inspections according to a preset parts inspection method (i.e., the visual inspection part in the diagram, which is also the vehicle assembly inspection method proposed in this application), and compares the inspection results with the configuration recorded in the standard parts list. After the inspection is completed, the framework software aggregates the inspection results via the aggregation workstation and sends them back to MOMO. This achieves online inspection, real-time uploading of results, and data recording through the platform software's data aggregation function, providing an effective entry point for NG traceability.

[0082] In this embodiment, in addition to forwarding data between MOMO and the software framework, the aggregation station can also record the forwarded data locally for verification when MOMO or the software framework malfunctions.

[0083] This concludes the discussion on... Figure 5 The description.

[0084] The following is through Figure 6A as well as Figure 6B The visual inspection portion mentioned above, namely the vehicle assembly inspection method proposed in this application, is described in its entirety.

[0085] Please refer to Figure 6A , Figure 6A A flowchart of the visual inspection process provided in the embodiments of this application.

[0086] like Figure 6A As shown, the visual inspection process may include the following steps: When the target vehicle arrives at the inspection station, vehicle images are acquired at each of the configured image acquisition points.

[0087] Obtain the configuration information for each image acquisition point. This configuration information includes the list of parts, the region to which the parts to be detected belong, and the detection type. For each image acquisition point, based on the region to which the part to be detected belongs, the region to be detected is determined from the vehicle image acquired at that image acquisition point. Each part area is inspected according to the inspection type of the part corresponding to each part area.

[0088] Specifically, if the part corresponding to the part area is a first specified part, a second specified part, or a third specified part, that is, it is necessary to determine whether the part exists, detect whether it is marked, or identify the part color, etc., based on traditional image processing algorithms, then according to... Figure 6B The detection methods corresponding to the conventional part detection model shown are used for subsequent processing, which will not be elaborated here.

[0089] If the part corresponding to the part region is the fourth specified part, that is, the shape of the part needs to be detected. Since the shape of the part is varied, the background is complex and difficult to describe with rules, a trained deep learning model, such as an object detection model, can be used for recognition.

[0090] Furthermore, the identification results of each part area are output.

[0091] Please refer to Figure 6B , Figure 6B This is a specific visual inspection flowchart provided for an embodiment of this application.

[0092] like Figure 6B As shown, at the start of the process, its input is from... Figure 6A The routine inspection parts list of the first designated part, the second designated part, and the third designated part obtained in the process continues as long as there are still uninspected parts in the list.

[0093] If any undetected parts exist in the above list, for each part, the corresponding Region of Interest (ROI) in the configuration information is determined, and the part region corresponding to that ROI in the acquired vehicle image is determined using a contour matching model. This ensures that subsequent detection can focus on the correct region (equivalent to performing a "part existence" detection). Then, based on the specific detection type of the part ("color mark detection" or "color detection"), the task is distributed to the corresponding detection submodule.

[0094] As one embodiment, the presence or absence of detection type is verified using a contour matching algorithm, and the color mark and color detection type are verified for consistency using a color extraction and RGB judgment algorithm. This application does not impose any limitations on this.

[0095] This concludes the discussion on... Figure 6B The description.

[0096] Please refer to Figure 7 , Figure 7 This is a flowchart of data loading and visual inspection provided for an embodiment of this application.

[0097] like Figure 7 As shown in the flowchart, this process describes an automated visual inspection workflow. When a vehicle enters the workstation, the system can automatically acquire vehicle model and location configuration information, as well as a parts list. Based on the location configuration information, it collects images, calls the matching parts detection methods for each part to perform inspection, and finally sends the inspection results back.

[0098] Specifically, MOMO provides initial instructions, which include vehicle information containing a specific model identifier. After receiving this information, the system interacts via the HTTP protocol, parses out the specific model to be inspected, and initiates a parts query with MOMO based on that model, thereby obtaining a precise list of parts to be inspected for that specific vehicle configuration.

[0099] Furthermore, encoders on the production line continuously provide encoder values ​​indicating the vehicle's real-time position, while sensors transmit status signals such as the vehicle's arrival. This real-time data, combined with pre-set acquisition point information, triggers a programmable logic controller (PLC) to generate precise control signals. These signals ultimately drive image acquisition equipment (such as an industrial camera) to perform a shooting action, acquiring an image of the vehicle at a specific location.

[0100] Simultaneously, the system reads detailed information for each part from the configuration tool, including the part number and its detection area (ROI) in the image, and prepares it for visualization on the interface. Furthermore, the system dynamically loads corresponding matching templates and color templates (such as Fmxml and Bin files) from the local template library to prepare data for subsequent part detection.

[0101] Once all image data and parameter templates are ready, the system will intelligently compare and analyze the point images collected on-site with the loaded templates and part parameters, and finally output structured configuration detection results to obtain the detection results for each part.

[0102] Subsequently, the system summarizes and integrates the independent inspection results of each part, and formats them into a unified and standardized data string. This formatted result is transmitted back to the MOMO system via HTTP protocol. Simultaneously, the system also performs a CSV save operation, persistently storing the complete inspection process data and results locally, forming data records available for quality traceability and analysis, thus achieving end-to-end digital management of the entire process.

[0103] This concludes the discussion on... Figure 7 The description.

[0104] In this embodiment, based on the analysis of inspection requirements, it is known that the inspection objects in the final assembly scenario are usually multi-point, multi-state, and non-fixed. Therefore, in order to improve the versatility of the inspection process, this application designs a data loading method for vehicle final assembly inspection. When different models and different optional vehicles enter the workstation, the matching template (i.e., the ROI area corresponding to the part) and the inspection model corresponding to the inspection type can be switched in real time according to the algorithm interface, so as to achieve visual inspection compatible with different identical objects and different subclasses, effectively avoiding the process of creating a large number of repetitive processes.

[0105] For example, after adding part A to the current parts list and configuring the ROI region and detection type for part A, the detection model corresponding to the ROI region and detection type can be directly stored locally on the electronic device. This allows for direct detection of part A by following the specified model. Figure 1 The assembly inspection method shown is used for testing.

[0106] In this embodiment, only the ROI region of each part and the corresponding detection model need to be stored for direct reuse. Figure 1 The new assembly and testing method significantly saves hardware resources, reducing deployment costs and computational requirements. It simplifies the previous approach of assigning one testing module to one part, allowing all parts to be connected to a single overall testing process. Combined with automotive configuration tools, configuration information is directly linked to the testing process, eliminating the need to manually create new testing processes. This greatly simplifies the addition of new models, parts, and configurations.

[0107] Meanwhile, this embodiment also implements the function of configuring detection parameters, enabling the switching and adjustment of detection parameters for image acquisition devices at different locations to accommodate influencing factors such as changes in ambient light and target scale. Addressing the impact of the complex environment and varied part states in the final assembly workshop on the detection algorithm, the deep learning detection module can automatically submit out-of-process (NG) data and images for review by MOMO, thereby improving model updates and iterations and achieving sustainable optimization of the model.

[0108] Please refer to Figure 8 , Figure 8This is a schematic diagram of a vehicle assembly and testing system provided in an embodiment of this application.

[0109] like Figure 8 As shown, the vehicle assembly and testing system includes: At least one electronic device for performing, such as Figure 1 The method described; The aggregation station is connected to each electronic device via HTTP communication. The aggregation station is used to collect the part inspection results sent by each electronic device for the target vehicle, and generate the final assembly inspection results of the target vehicle.

[0110] In this embodiment, the vehicle assembly and testing system may include multiple electronic devices, each of which may correspond to a production testing line, such as an exterior testing line, an interior testing line, a chassis testing line, etc. This application does not impose any limitations on this.

[0111] All of the aforementioned electronic devices are connected to the same data collection station to send the part inspection results to the data collection station, which then compiles and summarizes the final assembly inspection results of the target vehicle.

[0112] Please refer to Figure 9 , Figure 9 This is a structural diagram of a vehicle assembly and testing device proposed in an embodiment of this application. Figure 9 As shown, this device is applied to an electronic device. The device may include an acquisition unit 901, an identification unit 902, and a detection unit 903. Specifically, the device includes: The acquisition unit 901 is used to acquire vehicle images of the target vehicle acquired by the image acquisition device at the image acquisition point. The identification unit 902 is used to identify at least one part region from the vehicle image that matches the image acquisition point; different part regions correspond to different parts on the target vehicle; The detection unit 903 is used to detect each part area according to the part detection method matched to the part corresponding to that part area; wherein, the detection results of each part area are used to generate the final assembly inspection results of the target vehicle.

[0113] Optionally, the vehicle image includes: An image acquisition device fixed at a designated image acquisition point, capturing a vehicle image of the target vehicle when the target vehicle, located on a conveying device, is transported to a position matching the designated image acquisition point; and / or, An image acquisition device mounted on a robot's robotic arm, which acquires vehicle images of the target vehicle when the robotic arm moves along a specified trajectory to a specified location of the target vehicle, wherein the specified location includes at least the interior of the target vehicle; And / or, the detection unit is specifically used for: If the part corresponding to the part region is a first designated part, and the first designated part is a part in the target vehicle that needs to be detected to exist, then the region to which the part belongs is detected based on the trained contour detection model. If the part corresponding to the part region is the second designated part, and the second designated part is the part in the target vehicle that needs to be detected as to whether it is marked, then the part region is detected based on the trained mark detection model; If the part corresponding to the part region is a third designated part, and the third designated part is a part in the target vehicle whose color needs to be detected, then the part region is detected based on the trained color recognition model. If the part corresponding to the part region is the fourth designated part, and the fourth designated part is the part in the target vehicle whose shape needs to be detected, then the part region is detected based on the trained shape detection model. And / or, the electronic device is connected to an external aggregation station via HTTP communication; after detecting each part area according to the part detection method corresponding to the part in that part area, the detection unit is further configured to: The test results are sent to the aggregation industrial control computer in JSON string format via HTTP protocol, so that the aggregation industrial control computer can organize the test results of each part area and generate the final assembly test results of the target vehicle. And / or, the image acquisition point corresponding to the target vehicle, the part region matching the image acquisition point, and the part detection method matching the part corresponding to the part region are determined based on pre-stored configuration information; the configuration information is generated through the following steps: When a detection parameter configuration operation for the target vehicle model is detected, a first configuration interface is output. The first configuration interface is used to configure image acquisition points for the target vehicle corresponding to the target vehicle model. Upon detecting a configuration operation for the target image acquisition point in the first configuration interface, a second configuration interface is output. The second configuration interface is used to configure the part region to be detected and the corresponding part detection type for the target image acquisition point; different part detection types use different part detection methods. Based on the configuration operations received in the first configuration interface and the second configuration interface, configuration information matching the target vehicle corresponding to the target vehicle model is generated and stored; And / or, the first configuration interface includes a point list area and an overview area; the point list area records at least one image acquisition point configured for the target vehicle, and each image acquisition point is associated with device identification information, which indicates whether the image acquisition device at the image acquisition point is a fixed image acquisition device or an image acquisition device mounted on the robot's robotic arm; when any image acquisition point in the point list area is selected, the overview area displays the image currently acquired by the image acquisition device corresponding to that image acquisition point; The second configuration interface includes a parts list area and a parameter setting area; the parts list area includes the parts to be detected at the target image acquisition point, the part area to which each part belongs in the image acquired at the target image acquisition point, and the part detection type of each part; the parameter setting area includes parameter configuration information of at least one image acquisition device configured at the target image acquisition point, and the parameter configuration information includes at least one of exposure parameters, gain parameters, Gamma parameters, and process parameters.

[0114] This concludes the process. Figure 9 Description of the vehicle assembly and testing equipment.

[0115] This application also provides embodiments that... Figure 9 Hardware structure description of the illustrated device. This hardware structure is... Figure 10 The structure in the illustrated electronic device. Please refer to [link / reference]. Figure 10 , Figure 10 This is a structural diagram of an electronic device provided in an embodiment of this application. Figure 10 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0116] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the method disclosed in the above examples of this application.

[0117] For example, the aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For instance, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0118] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for vehicle final assembly testing, characterized in that, This method is applied to electronic devices, and the method includes: Obtain vehicle images of the target vehicle captured by the image acquisition device at the image acquisition point; Identify at least one part region from the vehicle image that matches the image acquisition point; different part regions correspond to different parts on the target vehicle; For each part area, the part area is inspected according to the part inspection method matched to the part corresponding to that part area; wherein, the inspection results of each part area are used to generate the final assembly inspection results of the target vehicle.

2. The method according to claim 1, characterized in that, The vehicle image includes: An image acquisition device fixed at a designated image acquisition point, capturing a vehicle image of the target vehicle when the target vehicle, located on a conveying device, is transported to a position matching the designated image acquisition point; and / or, An image acquisition device mounted on a robot's robotic arm acquires vehicle images of the target vehicle when the robotic arm moves along a specified trajectory to a specified location, wherein the specified location includes at least the interior of the target vehicle.

3. The method according to claim 1, characterized in that, The part region is inspected according to the part inspection method matched to the part corresponding to this part region, including: If the part corresponding to the part region is a first designated part, and the first designated part is a part in the target vehicle that needs to be detected to exist, then the region to which the part belongs is detected based on the trained contour detection model. If the part corresponding to the part region is the second designated part, and the second designated part is the part in the target vehicle that needs to be detected as to whether it is marked, then the part region is detected based on the trained mark detection model; If the part corresponding to the part region is a third designated part, and the third designated part is a part in the target vehicle whose color needs to be detected, then the part region is detected based on the trained color recognition model. If the part corresponding to the part region is the fourth designated part, and the fourth designated part is the part in the target vehicle whose shape needs to be detected, then the part region is detected based on the trained shape detection model.

4. The method according to claim 1, characterized in that, The electronic device is connected to an external aggregation workstation via HTTP communication. After detecting each part area according to the part detection method corresponding to that part area, the method further includes: The test results are sent to the aggregation industrial control computer in JSON string format via HTTP protocol. The aggregation industrial control computer then organizes the test results of each part area to generate the final assembly test results of the target vehicle.

5. The method according to claim 1, characterized in that, The image acquisition points corresponding to the target vehicle, the part regions matching the image acquisition points, and the part detection methods matching the parts corresponding to the part regions are determined based on pre-stored configuration information; the configuration information is generated through the following steps: When a detection parameter configuration operation for the target vehicle model is detected, a first configuration interface is output. The first configuration interface is used to configure image acquisition points for the target vehicle corresponding to the target vehicle model. When a configuration operation is detected for the target image acquisition point in the first configuration interface, a second configuration interface is output. The second configuration interface is used to configure the part area to be detected and the corresponding part detection type for the target image acquisition point. Different inspection methods are used for different types of parts. Based on the configuration operations received in the first configuration interface and the second configuration interface, configuration information matching the target vehicle corresponding to the target model is generated and stored.

6. The method according to claim 5, characterized in that, The first configuration interface includes a point list area and an overview area. The point list area records at least one image acquisition point configured for the target vehicle. Each image acquisition point is associated with device identification information, which indicates whether the image acquisition device at that point is a fixed image acquisition device or an image acquisition device mounted on the robot's robotic arm. When any image acquisition point in the point list area is selected, the overview area displays the image currently acquired by the image acquisition device corresponding to that point. The second configuration interface includes a parts list area and a parameter setting area; the parts list area includes the parts to be detected at the target image acquisition point, the part area to which each part belongs in the image acquired at the target image acquisition point, and the part detection type of each part; the parameter setting area includes parameter configuration information of at least one image acquisition device configured at the target image acquisition point, and the parameter configuration information includes at least one of exposure parameters, gain parameters, Gamma parameters, and process parameters.

7. A vehicle final assembly inspection system, characterized in that, include: At least one electronic device for performing the method as described in any one of claims 1 to 6; The aggregation station is connected to each electronic device via HTTP communication. The aggregation station is used to collect the part inspection results sent by each electronic device for the target vehicle, and generate the final assembly inspection results of the target vehicle.

8. A vehicle final assembly testing device, characterized in that, This device is used in electronic devices and includes: The acquisition unit is used to acquire vehicle images of the target vehicle captured by the image acquisition device at the image acquisition point. The identification unit is used to identify at least one part region from the vehicle image that matches the image acquisition point; different part regions correspond to different parts on the target vehicle; The detection unit is used to detect each part area according to the part detection method matched to the part corresponding to that part area; wherein, the detection results of each part area are used to generate the final assembly detection results of the target vehicle.

9. The apparatus according to claim 8, characterized in that, The vehicle image includes: An image acquisition device fixed at a designated image acquisition point, capturing a vehicle image of the target vehicle when the target vehicle, located on a conveying device, is transported to a position matching the designated image acquisition point; and / or, An image acquisition device mounted on a robot's robotic arm, which acquires vehicle images of the target vehicle when the robotic arm moves along a specified trajectory to a specified location of the target vehicle, wherein the specified location includes at least the interior of the target vehicle; And / or, the detection unit is specifically used for: If the part corresponding to the part region is a first designated part, and the first designated part is a part in the target vehicle that needs to be detected to exist, then the region to which the part belongs is detected based on the trained contour detection model. If the part corresponding to the part region is the second designated part, and the second designated part is the part in the target vehicle that needs to be detected as to whether it is marked, then the part region is detected based on the trained mark detection model; If the part corresponding to the part region is a third designated part, and the third designated part is a part in the target vehicle whose color needs to be detected, then the part region is detected based on the trained color recognition model. If the part corresponding to the part region is the fourth designated part, and the fourth designated part is the part in the target vehicle whose shape needs to be detected, then the part region is detected based on the trained shape detection model. And / or, the electronic device is connected to an external aggregation station via HTTP communication; after detecting each part area according to the part detection method corresponding to the part in that part area, the detection unit is further configured to: The test results are sent to the aggregation industrial control computer in JSON string format via HTTP protocol, so that the aggregation industrial control computer can organize the test results of each part area and generate the final assembly test results of the target vehicle. And / or, the image acquisition point corresponding to the target vehicle, the part region matching the image acquisition point, and the part detection method matching the part corresponding to the part region are determined based on pre-stored configuration information; the configuration information is generated through the following steps: When a detection parameter configuration operation for the target vehicle model is detected, a first configuration interface is output. The first configuration interface is used to configure image acquisition points for the target vehicle corresponding to the target vehicle model. Upon detecting a configuration operation for a target image acquisition point in the first configuration interface, a second configuration interface is output. The second configuration interface is used to configure the part region to be detected and the corresponding part detection type for the target image acquisition point; different part detection types employ different part detection methods. Based on the configuration operations received in the first configuration interface and the second configuration interface, configuration information matching the target vehicle corresponding to the target vehicle model is generated and stored; And / or, the first configuration interface includes a point list area and an overview area; the point list area records at least one image acquisition point configured for the target vehicle, and each image acquisition point is associated with device identification information, which indicates whether the image acquisition device at the image acquisition point is a fixed image acquisition device or an image acquisition device mounted on the robot's robotic arm; when any image acquisition point in the point list area is selected, the overview area displays the image currently acquired by the image acquisition device corresponding to that image acquisition point; The second configuration interface includes a parts list area and a parameter setting area; the parts list area includes the parts to be detected at the target image acquisition point, the part area to which each part belongs in the image acquired at the target image acquisition point, and the part detection type of each part; the parameter setting area includes parameter configuration information of at least one image acquisition device configured at the target image acquisition point, and the parameter configuration information includes at least one of exposure parameters, gain parameters, Gamma parameters, and process parameters.

10. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1 to 6.

Images