Vehicle final assembly offline quality inspection method, robot and system

Abstract

The invention relates to a vehicle final assembly offline quality inspection method, robot and system, and the method comprises the steps: an algorithm end detects whether a new vehicle enters a quality inspection station based on the monitoring data of a conveyor belt, and if yes, triggers a quality inspection task and obtains the initial position information of the vehicle relative to the robot; the agent side responds to the trigger signal of the quality inspection task, obtains the vehicle identification code of the vehicle, determines the vehicle type information of the vehicle according to the vehicle identification code, and sends the vehicle type information to the algorithm side; the algorithm end determines a plurality of quality inspection operation point locations in the process that the vehicle moves along with the conveying belt according to the vehicle type information and the initial position information, and controls the robot to execute quality inspection data acquisition operation at the plurality of quality inspection operation point locations in the process that the robot moves along with the vehicle to obtain quality inspection data; and uploading the quality inspection data to a cloud server through the agent end for defect detection. Based on the hierarchical collaborative architecture, the method can adapt to dynamic speed changes, intelligently match vehicle models for quality inspection, and improve the quality inspection efficiency and reliability.

Description

Technical Field

[0001] This application relates to the fields of intelligent manufacturing and industrial automation technology, and in particular to a method, robot and system for quality inspection of vehicles after final assembly. Background Technology

[0002] In the automobile manufacturing process, quality inspection at the final assembly line stage is a crucial step in ensuring vehicle compliance and brand reputation. With the increasing demand for personalization in the automotive consumer market, automobile manufacturers are generally adopting flexible production and mixed-flow assembly models. A single final assembly line often needs to process multiple different models, colors, and configurations of vehicles simultaneously, which places higher demands on the adaptability, precision, and efficiency of the final assembly line quality inspection process.

[0003] Currently, the following methods are mainly used for quality inspection of vehicles after final assembly: One method is manual visual inspection. This method relies on the experience and focus of quality inspectors, who check the vehicle's exterior and component assembly at the vehicle assembly line station. However, in high-paced production environments, manual inspection is susceptible to fatigue, changes in lighting, and subjective judgment, leading to significant fluctuations in the missed and false inspection rates. Furthermore, manual inspection has blind spots for defects on the bottom, top, or narrow corner areas of the vehicle, making comprehensive coverage difficult.

[0004] Second is the fixed vision inspection system. This type of system has multiple cameras fixedly installed around the workstation, capturing and analyzing vehicle images by triggering photo captures. While it improves the automation level of inspection to some extent, its flexibility is poor. When switching between different vehicle models on the production line, due to differences in vehicle dimensions and inspection points, the position, angle, and inspection program of the cameras often need to be readjusted and calibrated, resulting in long downtimes and making it difficult to adapt to the needs of flexible production. Furthermore, as the vehicle moves continuously along the conveyor belt, fixed cameras struggle to accurately capture predetermined inspection points, and image blurring is easily caused by fluctuations in conveyor belt speed, affecting inspection accuracy.

[0005] Thirdly, there are quality inspection systems based on mobile robots. In recent years, quality inspection solutions have emerged that utilize mobile robots equipped with vision sensors to inspect vehicles. These systems can autonomously move to the vicinity of the vehicles to be inspected and perform inspection tasks, offering a degree of flexibility compared to fixed workstations. However, most existing mobile robot quality inspection systems adopt a standalone architecture, with the robot acting only as an independent execution terminal, lacking efficient collaboration with the production management system. In practical applications, these systems struggle to dynamically adjust inspection points based on the actual location and movement status of the vehicles, typically requiring the vehicles to stop or pass through at low speed after the robot is in position, thus impacting production cycle time. Summary of the Invention

[0006] This application addresses the aforementioned shortcomings or deficiencies by providing a vehicle final assembly line quality inspection method, robot, and system. Based on a hierarchical collaborative architecture, this application can adaptively adapt to dynamic speed changes and intelligently match vehicle models for quality inspection, thereby improving inspection efficiency and reliability.

[0007] This application provides a vehicle final assembly line quality inspection method according to a first aspect. The method is applied to a robot, which includes an agent and an algorithm. The method includes: The algorithm detects whether a new vehicle has entered the quality inspection station based on the conveyor belt monitoring data from the depth camera. If so, it triggers the quality inspection task and obtains the initial position information of the vehicle relative to the robot. The agent responds to the trigger signal of the quality inspection task, obtains the vehicle identification code of the vehicle, determines the vehicle model information based on the vehicle identification code, and sends the model information to the algorithm. The algorithm determines multiple quality inspection points during the vehicle's movement along the conveyor belt based on vehicle model information and initial position information. It then controls the robot to sequentially perform quality inspection data collection operations at these multiple points as the vehicle moves alongside it, obtaining the quality inspection data. This data is then uploaded to the cloud server via an agent for defect detection.

[0008] According to the second aspect, this application provides a robot for quality inspection of vehicles after final assembly, comprising: The chassis is used to drive the robot to move within the work area; Dual robotic arms, mounted on the chassis; Depth camera, used to acquire depth images in the direction of the conveyor belt; Vision sensors are installed at the ends of each robotic arm; Vehicle identification number acquisition device, used to acquire the vehicle identification number of a vehicle; The agent, deployed in the first computing unit, is communicatively connected to the vehicle identification code acquisition device; and On the algorithm side, it is deployed in the second computing unit and communicates with the agent, depth camera, vision sensor, chassis and dual robotic arms. The algorithm and the agent are used to execute any of the vehicle final assembly quality inspection methods described in the above embodiments.

[0009] This application provides a vehicle final assembly line quality inspection system according to a third aspect, including: Cloud servers; and At least one robot for quality inspection after vehicle final assembly; The cloud server is used for: Send real-time speed data of the conveyor belt to the robot's agent; The robot receives quality inspection data uploaded by its agent, performs defect detection based on the data, and generates defect detection results.

[0010] In some embodiments, the cloud server is also used for: When the battery level of any robot falls below a preset threshold, a path is planned for the robot to reach a charging station, and another robot that meets preset conditions is assigned to take over the work. In the above embodiments of this application, an intelligent upgrade of vehicle final assembly line quality inspection is achieved by constructing a three-layer collaborative architecture of cloud-agent-algorithm. The cloud is centrally responsible for defect detection, undertaking heavy computation and model iteration to ensure detection accuracy and continuous evolution capability; the robot end is divided into agent end and algorithm end, realizing decoupling of task scheduling and real-time execution. This three-layer decoupling design enhances system robustness, supports independent algorithm iteration, rapid multi-machine expansion and centralized operation and maintenance, and effectively reduces the cost of intelligent transformation and maintenance of the production line.

[0011] This invention utilizes a communication-enabled agent and algorithm. The algorithm, based on conveyor belt monitoring data from a depth camera, actively detects vehicle entry and combines this with vehicle identification codes obtained from the agent to determine vehicle model information. This allows the robot to dynamically adjust its quality inspection points according to the specific vehicle model. Compared to traditional fixed-point detection methods, this approach adapts to the quality inspection needs of different vehicle models, avoiding missed or false inspections caused by mixed-model production lines. Furthermore, the algorithm determines multiple quality inspection points for the vehicle during its conveyor belt movement based on vehicle model information and initial position information, and controls the robot to sequentially collect data while following the vehicle. This dynamic detection breaks the limitations of traditional quality inspection where vehicles must be stationary or robots must be stationary for scanning. It synchronizes the quality inspection process with the continuous operation of the production line conveyor belt, significantly improving inspection efficiency and preventing production line congestion or reduced cycle time caused by quality inspection. Attached Figure Description

[0012] Figure 1 This is a flowchart of a vehicle final assembly line quality inspection method according to one or more embodiments of this application; Figure 2 This is a schematic diagram of the architecture of a vehicle final assembly line quality inspection system in one or more embodiments of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0014] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0015] In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0016] To address the shortcomings of related technologies, this application provides a vehicle final assembly line quality inspection method based on the first aspect. This method achieves an intelligent upgrade of vehicle final assembly line quality inspection by constructing a three-layer collaborative architecture of cloud-agent-algorithm. The cloud is centrally responsible for defect detection, undertaking heavy computation and model iteration to ensure detection accuracy and continuous evolution capability. The robot end is divided into agent end and algorithm end, realizing the decoupling of task scheduling and real-time execution. This three-layer decoupling design enhances system robustness, supports independent algorithm iteration, rapid multi-machine expansion, and centralized operation and maintenance, effectively reducing the cost of intelligent transformation and maintenance of production lines.

[0017] In some exemplary embodiments of this application, the method is applied to a robot, which includes an agent and an algorithm; such as Figure 1 As shown, the method includes steps S110-S140, and each step is described in detail below.

[0018] S110: The algorithm detects whether a new vehicle has entered the quality inspection station based on the conveyor belt monitoring data from the depth camera. If so, it triggers the quality inspection task and obtains the initial position information of the vehicle relative to the robot.

[0019] The algorithm side consists of embedded computing units (such as NVIDIA Jetson Orin) deployed on the robot body, used to perform latency-sensitive tasks such as real-time perception, motion planning and control.

[0020] A depth camera is an RGB-D (Red Green Blue-Depth) camera mounted on top of a robot. It is used to acquire real-time depth and color images along the direction of the conveyor belt and can perceive the three-dimensional spatial information of objects (such as vehicles) within the field of view.

[0021] Conveyor belt monitoring data refers to the sequence of depth images continuously acquired by depth cameras, used to monitor the traffic flow status on the conveyor belt.

[0022] A quality inspection station refers to a pre-set quality inspection area on the production line. When a vehicle enters this area, the robot begins to perform the quality inspection task.

[0023] Initial position information refers to the spatial position information of a specific part of the vehicle (such as the center point of the front of the vehicle) relative to the robot coordinate system at the trigger moment, usually including three-dimensional coordinates or spatial distance. The trigger moment refers to the time recorded when the quality inspection task is triggered.

[0024] In this step, after the algorithm starts, it continuously acquires depth images along the conveyor belt direction using a depth camera, forming a continuous monitoring data stream. For each frame of depth image, the algorithm uses image segmentation or object detection algorithms to identify vehicle targets within the field of view and counts the number of vehicles currently in the field of view. The algorithm records the number of vehicles corresponding to multiple consecutive frames in chronological order, forming a state sequence reflecting the continuous changes in the number of vehicles recently.

[0025] The algorithm is pre-defined with multiple pattern sets representing new vehicles entering the quality inspection station, such as the pattern of going from no vehicles to having vehicles, from one vehicle to two vehicles, or the pattern of the number of vehicles decreasing and then increasing. The algorithm dynamically compares the real-time generated state sequence with the predefined pattern set. When the change pattern of the state sequence matches any pattern in the pattern set, it is determined that a new vehicle has entered the quality inspection station, thus triggering the quality inspection task.

[0026] After triggering the quality inspection task, the algorithm locks the pixel coordinates of the center point of the new car's front end in the current frame's depth image. Combining the depth value corresponding to this pixel point output by the depth camera, as well as the pre-calibrated camera intrinsic parameters (including focal length and principal point coordinates) and camera extrinsic parameters (including the camera's rotation matrix and translation vector relative to the robot coordinate system), the algorithm transforms the pixel coordinates into three-dimensional spatial coordinates in the robot coordinate system, and then calculates the spatial distance between the center point of the car's front end and the origin of the robot coordinate system, as the initial position information.

[0027] Through the aforementioned deep visual monitoring and state sequence matching mechanism, this step can accurately trigger the quality inspection task the moment the vehicle enters the workstation, avoiding false triggering caused by changes in light and shadow or personnel movement, and laying a reliable time and position benchmark for subsequent follow-up data collection.

[0028] S120: The agent responds to the trigger signal of the quality inspection task, obtains the vehicle identification code of the vehicle, determines the vehicle model information based on the vehicle identification code, and sends the model information to the algorithm.

[0029] The agent is an industrial host (such as an x86 architecture industrial control computer) deployed on the robot body, used for non-real-time tasks such as data interaction with cloud servers, processing business logic, and managing hardware peripherals.

[0030] The Vehicle Identification Number (VIN) is a unique identifier for a vehicle and is usually stored in an ultra-high frequency RFID (Radio Frequency Identification) electronic tag installed on the vehicle.

[0031] Vehicle information includes parameters related to quality inspection operations, such as vehicle model, overall length, paint color, and configuration of inspection points (e.g., the ratio of the front, middle, and rear sections of the vehicle).

[0032] In this step, when the algorithm triggers a quality inspection task, it sends a task trigger signal to the agent. The agent responds to this signal by controlling the UHF RFID scanner mounted on the robot to emit radio frequency signals and scan the RFID tags at preset locations on the vehicle. Once the RFID scanner successfully reads the vehicle identification code stored in the tag, the agent uses this code as an index to retrieve the complete vehicle model information, either locally or by querying the cloud. This model information may include parameters such as the vehicle's total length, paint color type, and preset detection point offset ratio.

[0033] After obtaining the vehicle model information, the agent encapsulates the vehicle identification code and model information and sends them to the algorithm through the robot's internal communication protocols, such as ROS2 (Robot Operating System 2) or MQTT (Message Queuing Telemetry Transport), for use by the algorithm in subsequent steps.

[0034] This step enables the automatic acquisition of vehicle identity information and the adaptive loading of vehicle model parameters, eliminating the need for manual data entry or scanning. It provides a data foundation for subsequent vehicle model-based differentiated testing and establishes a starting point for the association between vehicle identification codes and quality inspection data.

[0035] S130: The algorithm determines multiple quality inspection work points during the vehicle's movement along the conveyor belt based on the vehicle model information and initial position information, and controls the robot to perform quality inspection data collection operations at multiple quality inspection work points in sequence while following the vehicle to obtain quality inspection data.

[0036] Quality inspection work points refer to multiple preset inspection positions along the length of the vehicle, usually corresponding to the front, middle and rear sections of the vehicle, used to collect images or sensor data from different parts of the vehicle.

[0037] Following vehicle movement refers to a robot maintaining a relative position with a vehicle on a conveyor belt through autonomous motion control, thereby achieving stable data collection during vehicle movement.

[0038] In this step, the algorithm first extracts the total length of the vehicle from the vehicle model information received from the agent. Then, based on the initial position information obtained in step S110 and the total length of the vehicle, it calculates the target distance values ​​corresponding to multiple quality inspection work points.

[0039] After determining the target distance values ​​for each quality inspection work point, the algorithm begins to execute follow-up movement and data acquisition control. The algorithm receives conveyor belt speed data from the cloud server in real time, accumulates the speed data over time using an integral algorithm, and calculates the real-time displacement of the vehicle as it moves with the conveyor belt from the trigger moment.

[0040] The algorithm can combine chassis movement with integral waiting to complete the quality inspection data collection operation. Various strategies can be employed for chassis movement. For example, a following strategy can be used: the robot chassis moves forward at a speed matching the conveyor belt speed, maintaining a roughly constant position relative to the vehicle. During this process, the algorithm continuously compares the real-time displacement with the target distance value corresponding to the current quality inspection work point for which the quality inspection data collection operation is to be performed. When the real-time displacement reaches or exceeds the target distance value, the algorithm triggers the vision sensors installed at the ends of the robot's dual robotic arms to perform the quality inspection data collection operation, acquiring images or sensor data of the current quality inspection work point. Another example is to first control the robot chassis to move to the vicinity of the current quality inspection work point at a relatively fast speed, then continuously compare the real-time displacement with the target distance value corresponding to the current quality inspection work point for which the quality inspection data collection operation is to be performed; when the real-time displacement reaches or exceeds the target distance value, the quality inspection data collection operation is executed.

[0041] After completing the data collection at one quality inspection point, the algorithm will take the next quality inspection point as the current point to be executed and repeat the above comparison and triggering process until all quality inspection points have completed data collection.

[0042] This step achieves dynamic and accurate data acquisition while the vehicle is continuously moving by combining integral following and point pre-planning. The robot can complete multi-point data acquisition without the vehicle stopping or slowing down, ensuring the production line cycle time. At the same time, point planning based on vehicle model information enables the system to adapt to the inspection requirements of vehicles of different lengths, and can handle vehicle model changes without hardware adjustments.

[0043] S140: The algorithm uploads the quality inspection data to the cloud server via the agent for defect detection.

[0044] Quality inspection data includes images and sensor data collected at each quality inspection point, as well as metadata related to the collection, such as the collection time and robot status.

[0045] Cloud servers are server clusters deployed in remote or on-site data centers, equipped with high-performance GPU (Graphics Processing Unit) computing resources, used to run large-scale AI (Artificial Intelligence) defect detection models.

[0046] After the algorithm completes data collection at all quality inspection points, it packages the collected image or sensor data along with metadata such as vehicle identification number (VIN), vehicle model information, collection time, and robot self-inspection status into an encrypted data packet. The algorithm then sends this encrypted data packet to the agent via the robot's internal communication protocol. The robot self-inspection status indicates the health and operational status of each key subsystem before the robot performs the quality inspection task, including the robotic arm, camera, chassis, sensors, power supply, and communication system. The self-inspection status is represented by enumerated values ​​or status codes, such as "OK" (normal), "WARN" (warning), "ERROR" (error), "CALIB" (calibrating), and "MAINT" (maintenance mode), used to ensure data collection reliability, trigger anomaly handling, and monitor the task loop. After image capture, this status, along with the image, VIN, vehicle model information, and collection time, is encapsulated into an encrypted data packet, verified by the agent, and uploaded to the cloud server, thus achieving a closed-loop quality inspection process.

[0047] After receiving the data packet, the agent first performs an integrity check to ensure that no frames are lost or corrupted during transmission. Once the check passes, the agent uploads the data packet to the cloud server via industrial Ethernet or a 5G wireless network.

[0048] After receiving the data packet, the cloud server parses the quality inspection data and vehicle identification number (VIN), and then calls a pre-trained AI defect detection model to analyze the data. The AI ​​model can identify scratches, dents, and paint defects on the vehicle body surface, as well as the assembly status and model markings of components. For detection results with low confidence levels from the AI ​​model, the cloud server can automatically push them to a human quality inspection station for remote secondary verification. The cloud server then associates the final defect detection results with the VIN and stores them in the production quality database, forming a complete quality traceability record. Based on this, precise binding and closed-loop processing of quality inspection data and vehicle identification are achieved, leveraging the advantages of large-scale cloud computing power for AI defect detection and establishing a complete data traceability link from production to quality inspection, providing reliable data support for quality analysis and process improvement.

[0049] This application embodiment enables adaptive multi-point quality inspection based on vehicle model information in dynamic scenarios where vehicles are continuously moving, through layered collaboration between the cloud, algorithm, and agent. It also achieves accurate correlation and closed-loop processing of quality inspection data in the cloud through vehicle identification codes, effectively improving quality inspection accuracy, production flexibility, and data traceability.

[0050] In some embodiments, the algorithm detects whether a new vehicle has entered the quality inspection station based on conveyor belt monitoring data from a depth camera, including: the algorithm identifies changes in the number of vehicles within the field of view of the depth camera based on a depth image generated by the depth camera in the direction of the conveyor belt, and generates a traffic flow status sequence; the traffic flow status sequence is compared with a predefined pattern set, which includes multiple patterns representing changes in the number of vehicles entering the quality inspection station; when the traffic flow status sequence matches any pattern in the predefined pattern set, it is determined that a new vehicle has entered the quality inspection station.

[0051] A depth image is a two-dimensional image output by a depth camera. Each pixel stores the distance information from that point to the camera, and is used to perceive the three-dimensional spatial position of objects within the field of view.

[0052] Traffic flow status sequence is a trajectory of the number of vehicles in the field of view recorded in a continuous series of depth images in chronological order, used to reflect the dynamic changes in traffic flow recently.

[0053] The predefined pattern set is a collection of multiple pre-set vehicle quantity change patterns. Each pattern corresponds to a typical scenario of a new vehicle entering the quality inspection station, such as from no vehicle to vehicle, from one vehicle to two vehicles, or the number of vehicles decreasing and then increasing.

[0054] The vehicle quantity change pattern refers to the specific pattern of vehicle quantity change over time, expressed in the form of a state sequence. For example, "1→2" means that the number of vehicles in the depth camera's field of view changes from one to two, "2→1→2" means that the number of vehicles changes from two to one and then back to two, "N→1" means that the number of vehicles in the depth camera's field of view changes from zero (i.e. no vehicles) to one, and "1→N→1" means that the number of vehicles changes from one to zero and then back to one.

[0055] The algorithm continuously acquires depth images along the conveyor belt direction using a depth camera, forming a continuous data stream. For each frame of depth image, the algorithm uses image segmentation algorithms or deep learning-based object detection models to identify and count the pixel regions in the image that belong to vehicle targets, thereby obtaining the number of vehicles in the current field of view.

[0056] The algorithm maintains a fixed-length queue to store the vehicle count values ​​corresponding to the most recent few image frames. After processing each depth image frame, the algorithm adds the currently counted vehicle count to the end of the queue and removes the oldest data from the head of the queue, ensuring that the queue always maintains a sliding time window reflecting the continuous changes in the number of vehicles recently. This queue is the traffic flow state sequence.

[0057] The algorithm pre-sets multiple modes to represent new vehicles entering the quality inspection station. These modes are pre-defined based on the vehicle entry patterns in actual production scenarios, including but not limited to the following typical modes: from no vehicle to vehicle in view (N→1); from one vehicle to two vehicles in view (1→2); from two vehicles to one vehicle and then back to two vehicles in view (2→1→2); from one vehicle to no vehicle and then back to one vehicle in view (1→N→1). These modes cover various actual working conditions such as new vehicles entering the view from behind the conveyor belt, new vehicles entering after the previous vehicle has driven out, and vehicles temporarily lost due to brief obstruction and then restored.

[0058] The algorithm dynamically compares the real-time generated traffic flow state sequence with each pattern in the predefined pattern set. During the comparison, the algorithm determines whether the trajectory of the traffic flow state sequence matches the state transition path of a certain pattern. When the pattern of the traffic flow state sequence completely matches any pattern in the predefined pattern set, the algorithm confirms that a new vehicle has entered the quality inspection station, thus triggering the quality inspection task.

[0059] Through the detection mechanism based on matching vehicle flow state sequences with predefined pattern sets, this embodiment can accurately identify the timing of new vehicles entering the workstation, effectively avoiding false triggers caused by changes in lighting, brief personnel movement, or glare interference. Compared to traditional methods using single-frame detection or simple threshold judgment, this embodiment fully utilizes information in the time dimension through matching continuous state sequences, significantly improving the reliability and anti-interference capability of triggering, and providing a reliable time reference for the accurate execution of subsequent quality inspection operations.

[0060] In some embodiments, the algorithm determines multiple quality inspection work points during the vehicle's movement along the conveyor belt based on vehicle model information and initial position information. This includes: obtaining the vehicle's body length based on the vehicle model information; determining the initial distance between the vehicle's front end and the robot based on the initial position information; and determining the target distance values ​​corresponding to each of the multiple quality inspection work points based on the vehicle body length and the initial distance. These multiple quality inspection work points correspond to the front, middle, and rear sections of the vehicle, respectively.

[0061] Vehicle length refers to the total length of the vehicle, the distance from the frontmost point of the vehicle to the rearmost point of the vehicle. This parameter is stored in the vehicle model information and can be used to calculate the distribution of each quality inspection point along the length of the vehicle.

[0062] The front, middle, and rear sections of a vehicle refer to three sections divided along its length. The front section corresponds to the front area of ​​the vehicle and typically includes the front bumper, headlights, and hood; the middle section corresponds to the middle area of ​​the vehicle and typically includes the doors, B-pillars, and side mirrors; the rear section corresponds to the rear area of ​​the vehicle and typically includes the rear bumper, taillights, and trunk lid. The proportions of each section can be flexibly set according to the inspection requirements.

[0063] In this embodiment, the algorithm extracts the vehicle body length parameter of the current vehicle from the vehicle model information sent by the agent. Vehicle body length is usually measured in millimeters or meters, and the body length varies between different vehicle models, such as sedans, SUVs (Sport Utility Vehicles), and MPVs (Multi-Purpose Vehicles).

[0064] Simultaneously, the algorithm determines the initial distance between the vehicle's front and the robot based on the initial position information. This initial distance is the spatial distance between the center point of the vehicle's front and the origin of the robot's coordinate system at the moment of triggering. This initial distance reflects the relative positional relationship between the robot and the vehicle's front at the instant the vehicle enters the workstation.

[0065] In some embodiments, the initial distance calculation process includes: the algorithm acquiring a depth image containing the front of the vehicle using a depth camera; identifying the pixel coordinates of the center point of the front of the vehicle from the depth image and obtaining the depth value corresponding to the center point of the front of the vehicle; converting the pixel coordinates and depth value into three-dimensional spatial coordinates in the robot coordinate system according to pre-calibrated camera intrinsic and extrinsic parameters; and calculating the spatial distance between the center point of the front of the vehicle and the origin of the quality inspection robot coordinate system based on the three-dimensional spatial coordinates, as the initial distance.

[0066] A depth image is a two-dimensional image output by a depth camera. Each pixel stores the distance information (depth value) from that point to the camera, used to perceive the three-dimensional spatial position of objects within the field of view. By aligning a depth image with a color image, both the color information and spatial distance information of each pixel can be obtained simultaneously.

[0067] The center point of the vehicle's front end refers to the geometric center of the front area, typically located in the center of the front bumper, near the vehicle logo, or in the center of the front grille. This point serves as a reference point for the vehicle's position, used to calculate the relative distance between the robot and the vehicle.

[0068] Pixel coordinates refer to the position of the center point of the vehicle's front end in the image, which can be represented by two-dimensional coordinates (u, v), with the unit being pixels. Pixel coordinates reflect the projected position of the center point of the vehicle's front end within the camera's field of view.

[0069] Depth values ​​refer to the distance values ​​stored at corresponding pixels in a depth image, representing the physical distance from that point to the camera, usually measured in millimeters or meters. Raw depth values ​​need to be multiplied by a scaling factor to convert them to true physical depth.

[0070] Camera intrinsics are parameter matrices that describe the internal optical characteristics of a camera, including the focal length (f_x, f_y) and principal point coordinates (c_x, c_y). The intrinsic matrix is ​​used to project 3D points in the camera coordinate system onto the image plane, or conversely, to convert pixel coordinates into direction vectors in the camera coordinate system.

[0071] The camera extrinsic parameters 's' describe the pose relationship between the camera coordinate system and the robot coordinate system, including the rotation matrix R and the translation vector T. These extrinsic parameters are used to transform the 3D coordinates from the camera coordinate system to the robot coordinate system.

[0072] A robot coordinate system is a coordinate system established with the robot body as a reference. It typically has its origin at the center of the robot's chassis or a fixed point, with the X-axis pointing forward, the Y-axis pointing to the left, and the Z-axis pointing vertically upward. The robot coordinate system serves as the reference for the robot's motion control and position awareness.

[0073] Three-dimensional spatial coordinates refer to the spatial position of the center point of the robot's front end in the robot's coordinate system, which can be represented by (X_robot, Y_robot, Z_robot), with units of millimeters or meters. This coordinate system reflects the precise spatial position of the robot's front end relative to the robot.

[0074] Spatial distance refers to the Euclidean distance between the center point of the vehicle's front and the origin of the robot's coordinate system.

[0075] After the algorithm confirms that the new vehicle has entered the quality inspection station through the vehicle front detection mechanism, the algorithm needs to accurately obtain the initial position information of the vehicle relative to the robot, specifically by performing the following operations: First, the algorithm acquires the depth image captured by the depth camera at the current moment. This depth image is aligned with the color image and contains the three-dimensional spatial information of the vehicles within the field of view.

[0076] Secondly, the algorithm identifies the location of the center point of the vehicle's front end from the depth image. The center point can be located in the color image using a pre-trained deep learning object detection model (such as YOLO or Faster R-CNN), and then the center point of the detection box is mapped to the corresponding pixel location in the depth image; alternatively, feature matching can be performed directly on the depth image to identify the geometric feature points of the vehicle's front end region. The algorithm obtains the pixel coordinates (u, v) of the center point of the vehicle's front end and extracts the corresponding raw depth value d_raw from the depth image.

[0077] The algorithm converts the original depth value into the true physical depth z_depth based on the scale factor of the depth camera. The calculation formula is as follows: z_depth=d_raw×scale_factor.

[0078] This physical depth represents the distance from the center point of the vehicle's front to the optical center of the camera along the optical axis.

[0079] Subsequently, the algorithm performs coordinate transformation using the pre-calibrated camera intrinsic matrix K and extrinsic matrix (R, T). Here, R is the rotation matrix used to rotate the camera coordinate system to the robot coordinate system, and T is the translation vector representing the displacement between the camera origin and the robot coordinate system origin. Furthermore, the camera intrinsic matrix K is: ; Where f_x and f_y represent the focal length, and c_x and c_y represent the principal point coordinates of the depth image.

[0080] The algorithm, based on the pinhole camera model, converts the pixel coordinates (u, v) and physical depth z_depth into three-dimensional coordinates (X_c, Y_c, Z_c) in the camera coordinate system, where: ; ; Z_c = .

[0081] Then, the algorithm uses the rotation matrix R and translation vector T from the camera's extrinsic parameters to transform the coordinates from the camera coordinate system to the robot coordinate system, as shown in the following formula: ; At this point, the algorithm has obtained the three-dimensional spatial coordinates (X_robot, Y_robot, Z_robot) of the center point of the vehicle's front end in the robot coordinate system.

[0082] Finally, the algorithm calculates the spatial distance D from the center point of the vehicle's front end to the origin of the robot's coordinate system based on the three-dimensional spatial coordinates. The calculation formula is as follows: ; The algorithm then stores this distance value for use in planning and calculating subsequent quality inspection points.

[0083] The initial distance calculation method based on depth images and coordinate transformation described above enables precise measurement of vehicle positions in complex industrial environments. Compared to traditional methods relying on external sensors or manual measurement, this embodiment utilizes the robot's built-in depth camera to acquire the three-dimensional spatial coordinates of the vehicle's center point from a single frame image, eliminating the need for additional external positioning equipment. Precise calibration of the camera's intrinsic and extrinsic parameters converts pixel-level image information into millimeter-level spatial distances, providing a high-precision initial reference for subsequent integral following and point planning. This method is particularly suitable for production scenarios where vehicles dynamically enter workstations and robot positions may change. It adaptively acquires the initial positional relationship for each quality inspection task, laying a reliable foundation for precise operations in dynamic environments.

[0084] After obtaining the initial distance, the algorithm calculates the theoretical distance values ​​for multiple quality inspection points corresponding to the front, middle, and rear sections of the vehicle, based on the vehicle length and the initial distance. These theoretical distance values ​​serve as the target distance values ​​for each quality inspection point. Specifically, the algorithm first determines the offset of each section along the vehicle length. For example, for the quality inspection point corresponding to the front section, the offset can be set to one-sixth of the vehicle length, indicating that the point is located one-sixth of the vehicle length behind the front of the vehicle; for the quality inspection point corresponding to the middle section, the offset can be set to one-half of the vehicle length, indicating that the point is located in the middle of the vehicle; and for the quality inspection point corresponding to the rear section, the offset can be set to five-sixths of the vehicle length, indicating that the point is located five-sixths of the vehicle length behind the front of the vehicle.

[0085] Then, the algorithm adds the aforementioned offsets to the initial distances to obtain the target distance values ​​corresponding to each quality inspection point. Taking an initial distance of D and a vehicle length of L as an example, the target distance value for the front point is D+L / 6, the target distance value for the middle point is D+L / 2, and the target distance value for the rear point is D+5L / 6. The above proportional coefficients are only examples. In practical applications, the offset ratio of each segment can be flexibly adjusted according to the inspection focus of different vehicle models and the field of view of the vision sensor. For example, the front point can be set to L / 5, the middle point to L / 2, and the rear point to 4L / 5, etc.

[0086] The algorithm stores the calculated distance values ​​of each target locally for subsequent trigger judgment during the following movement process.

[0087] By employing the point planning method based on vehicle body length and initial distance described above, this embodiment achieves adaptive determination of quality inspection operation points. When the production line switches to different vehicle models, the system can automatically adjust the target distance values ​​corresponding to each quality inspection operation point by obtaining the current vehicle body length, without requiring any adjustments to the robot hardware or sensor installation positions. This allows the method to flexibly adapt to the inspection needs of multi-model co-production in mixed-flow assembly mode, significantly reducing preparation time and equipment debugging costs during model switching. Simultaneously, by planning the front, middle, and rear sections of the vehicle separately, this method can achieve comprehensive coverage of all key areas of the vehicle, avoiding blind spots caused by unreasonable point placement.

[0088] In some embodiments, determining the target distance value corresponding to each of the multiple quality inspection work points based on the vehicle body length and initial distance includes: determining the theoretical distance value corresponding to each of the multiple quality inspection work points corresponding to the front, middle and rear sections of the vehicle based on the vehicle body length and initial distance; and compensating the theoretical distance value corresponding to each quality inspection work point based on the weighted average of the system delay time and the recent speed of the conveyor belt to obtain the target distance value corresponding to each quality inspection work point.

[0089] The theoretical distance value refers to the distance between each quality inspection point and the robot's starting position under ideal conditions (without considering system latency). This value is calculated based on the initial distance and the robot's length (see the description in the above embodiment for the specific calculation method) and is used to characterize how far the robot needs to move to align with the target inspection area.

[0090] System latency refers to the time difference between the algorithm issuing a data acquisition command and the actual execution of the data acquisition operation. This latency mainly includes communication transmission time, sensor response time, and mechanical structure action time. System latency can be obtained through pre-calibration or real-time measurement.

[0091] The weighted average of the recent conveyor belt speed is calculated by assigning different weights to the conveyor belt speed data from the most recent frames, with frames having larger weights corresponding to more recent time points. This weighted average is used to estimate the distance the vehicle travels within the system delay time, and the weight settings make the estimation result closer to the current actual speed change trend.

[0092] In this embodiment, the target distance value refers to the distance value actually used to trigger the judgment after compensation. This value is equal to the theoretical distance value minus the estimated movement distance of the vehicle within the system delay time, and is used to trigger the acquisition command in advance to compensate for the position deviation caused by the system delay.

[0093] Specifically, the algorithm first determines the theoretical distance values ​​corresponding to each quality inspection point based on the vehicle length and initial distance, following the aforementioned method. Let the theoretical distance value for the quality inspection point corresponding to the front section of the vehicle be D_theory1, the theoretical distance value for the quality inspection point corresponding to the middle section be D_theory2, and the theoretical distance value for the quality inspection point corresponding to the rear section be D_theory3. These theoretical distance values ​​represent the distance the robot needs to move under ideal, zero-delay conditions.

[0094] However, in real-world industrial environments, there is an inherent system delay between the algorithm issuing the acquisition command and the camera's actual exposure. This delay can originate from multiple factors, including communication transmission, sensor triggering, and image exposure time. During this delay, the vehicle continues to move along the conveyor belt. If the theoretical distance is used directly as the trigger condition, the actual acquired position will be further back than the predetermined position, resulting in a positional deviation.

[0095] To address this issue, this embodiment introduces a dynamic compensation mechanism. The algorithm continuously acquires the current system delay time Δt_delay, which can be obtained through system calibration or real-time measurement. Simultaneously, the algorithm continuously receives conveyor belt speed data from the cloud server, recording the speed values ​​v_1, v_2, ..., v_5 for the most recent few frames (e.g., 5 frames), and assigns weights w_1, w_2, ..., w_5 to each frame, with frames closer to the current time receiving greater weights. The algorithm calculates the weighted average of the recent conveyor belt speeds, V_avg. ; For the first Frame distance value, For the first Weights of frame distance values.

[0096] This weighted average is used to estimate the vehicle's travel distance during the system delay time. The algorithm compensates for the theoretical distance values ​​of each quality inspection point to obtain the compensated target distance value: ; Where D_theory is the theoretical distance value of a certain quality inspection operation point (such as D_theory1, D_theory2, D_theory3), V_avg·Δt_delay is the estimated movement distance of the vehicle within the system delay time, and D_target is the target distance value used for actual triggering judgment after compensation.

[0097] Taking the mid-section point as an example, let the theoretical distance be D_theory2, then the compensated target distance is D_target2 = D_theory2 - V_avg·Δt_delay. During subsequent following movement, the algorithm compares the real-time displacement with this target distance value. When the real-time displacement reaches or exceeds the target distance value, a data acquisition command is issued. Because there is a system delay before data acquisition is actually performed, the vehicle has just moved to the position corresponding to the theoretical distance value, thus achieving precise compensation for the system delay.

[0098] The algorithm performs the above compensation calculation on each quality inspection point to obtain the target distance value corresponding to each point, and stores it locally for subsequent trigger judgment.

[0099] Through the aforementioned dynamic compensation mechanism, this embodiment effectively eliminates the impact of system latency on the accuracy of the acquisition position. In the event of conveyor belt speed fluctuations or changes in system latency, the compensation mechanism can dynamically adjust the trigger threshold based on the weighted average of the current speed, ensuring that the acquisition time accurately corresponds to the predetermined theoretical position. Compared to traditional triggering methods that do not consider latency, this embodiment significantly improves the accuracy of the acquisition position in dynamic scenarios, especially on production lines with high-speed movement or frequent speed changes. It effectively avoids problems such as image blurring and detection area offset caused by latency, ensuring the accuracy and reliability of quality inspection data.

[0100] In some embodiments, controlling the robot to sequentially perform quality inspection data collection operations at multiple quality inspection work points while following the vehicle includes: the algorithm determining the target quality inspection work point from multiple quality inspection work points to be used for the quality inspection data collection operation; controlling the robot to move the movement distance value corresponding to the target quality inspection work point; the movement distance value corresponding to the target quality inspection work point being less than the target distance value corresponding to the target quality inspection work point; based on the real-time conveyor belt speed issued by the cloud server, calculating the real-time displacement generated by the vehicle moving with the conveyor belt from the triggering time of the quality inspection task using an integral algorithm; when the real-time displacement reaches the target distance value corresponding to the target quality inspection work point, controlling the robot to perform the quality inspection data collection operation; determining whether the quality inspection data collection operation has been completed at all quality inspection work points, and if not, returning to the step of determining the target quality inspection work point from multiple quality inspection work points to be used for the quality inspection data collection operation.

[0101] The target quality inspection point refers to the point among multiple quality inspection points where the quality inspection data collection operation is currently to be performed. The system processes each point sequentially according to a preset order (e.g., from front to back).

[0102] The travel distance refers to the distance the robot chassis actively moves forward. This value is determined by the algorithm based on the target distance of the target quality inspection work point, and is usually set to be less than the target distance to reserve a certain amount of travel for integration while waiting for triggering.

[0103] The target distance value for each quality inspection point can be the theoretical distance value for each quality inspection point, or the distance value obtained by compensating for the theoretical distance value. The calculation method of this value has been described in detail in the above embodiments. Its physical meaning is that the robot can only perform the data collection operation when the distance the vehicle moves with the conveyor belt from the triggering time reaches this value.

[0104] The integral algorithm refers to an algorithm that calculates the displacement of a vehicle as it moves along the conveyor belt by integrating the conveyor belt speed data over time.

[0105] Real-time displacement refers to the cumulative displacement of the vehicle as it moves along the conveyor belt from the moment of triggering to the present moment. This value is calculated in real time using an integral algorithm and is used for comparison with the target distance value.

[0106] Real-time displacement can be calculated using the following formula: ; in, This refers to the real-time conveyor belt speed transmitted from the cloud, Δt refers to the sampling interval, and n refers to the number of times the conveyor belt speed is received during the shooting process. This refers to the speed of the conveyor belt received on the j-th receiving trip. This refers to real-time movement and displacement. Indicates the current moment. Indicates the trigger time.

[0107] In this embodiment, the algorithm first obtains multiple identified quality inspection operation points and their corresponding target distance values. Following a preset order (usually from the beginning to the end), the algorithm determines the target quality inspection operation point from these multiple points to be used for the current quality inspection data collection operation.

[0108] After determining the target quality inspection operation point, the algorithm side controls the robot chassis to move forward. The distance value of the chassis movement is a preset movement distance value, which is set to be less than the target distance value corresponding to the target quality inspection operation point. For example, if the target distance value is D_target, the robot chassis first actively moves forward by D_move_robot, and D_move_robot < D_target. This design allows the robot to reserve a certain travel distance for subsequent integral waiting to be triggered after actively moving a certain distance, avoiding the situation where the vehicle has not arrived yet when the robot reaches the target position prematurely.

[0109] While the robot chassis is moving, the algorithm side activates the speed integrator and receives the conveyor belt speed data v_i sent by the cloud server in real time. The algorithm side integrates and accumulates the speed data at a preset sampling interval Δt (e.g., 0.1 second) to calculate the real-time moving displacement generated by the vehicle moving along the conveyor belt from the trigger moment.

[0110] The algorithm side continuously compares the real-time moving displacement with the target distance value corresponding to the target quality inspection operation point dynamically. When the real-time moving displacement reaches or exceeds the target distance value, the algorithm side triggers the execution of the quality inspection data acquisition operation at the current target quality inspection operation point. At this time, the robot chassis has moved the preset movement distance value, and the vehicle has also moved a corresponding distance. The superposition of the two makes the acquisition device of the robot exactly对准 the预定检测区域 on the vehicle.

[0111] After completing the quality inspection data acquisition operation at the current target quality inspection operation point, the algorithm side determines whether the data acquisition has been completed at all quality inspection operation points. If there are still uncompleted points, it returns to the step of determining the target quality inspection operation point for which the quality inspection data acquisition operation is to be performed from multiple quality inspection operation points, takes the next quality inspection operation point as the new target point, and repeats the above movement and trigger process until the data acquisition operations at all quality inspection operation points have been completed.

[0112] Through the above control method of segmented movement + integral trigger, this embodiment achieves precise acquisition of multiple quality inspection operation points during the continuous movement of the vehicle. On the one hand, the robot actively moves a part of the distance, reducing the travel distance that the robot needs to follow, and reducing energy consumption and mechanical wear; on the other hand, by integrating the algorithm to accumulate the vehicle displacement in real time and triggering the acquisition when the preset threshold is reached, it ensures the precise correspondence between the acquisition moment and the predetermined position. Compared with the traditional method that requires the vehicle to stop or the robot to follow precisely throughout the process, this embodiment takes into account both the production rhythm and the acquisition accuracy, especially suitable for dynamic scenarios where the conveyor belt speed fluctuates, effectively improving the stability and reliability of the quality inspection operation.

[0113] In some embodiments, the method further includes: the algorithm controls the robot to move to the door detection position to perform a door image acquisition operation; the acquired door image is identified based on the door state detection model to determine whether the door is in a closed state; if the door is in an open state, the quality inspection task is stopped, the robot is controlled to perform an emergency avoidance operation and report the abnormality.

[0114] The door detection position refers to the pre-set pose of the robotic arm on the robot, which facilitates image acquisition of the side door area of ​​a vehicle. This position typically positions the vision sensor mounted at the end of the robotic arm directly facing the side of the vehicle, enabling clear acquisition of a complete image of the door.

[0115] The door image acquisition operation refers to the operation of controlling the robot to move to the door detection position and then taking pictures or videos of the side door area of ​​the vehicle through a vision sensor.

[0116] A car door state detection model is a pre-trained deep learning model used to identify whether a car door is closed or open based on captured images. This model can employ object detection network architectures such as YOLO or Faster R-CNN, and is trained using a large number of labeled car door images.

[0117] Taking the YOLO11 architecture as an example, the training process may include the following steps: (1) Data collection Collect images of vehicle doors from different vehicles on the final assembly line under different lighting conditions; Each door image is labeled with a category, such as Open or Closed. (2) Training configuration The loss function is chosen to be bounding box regression loss (CIoU Loss) and class prediction loss (such as cross-entropy or BCE). The total loss = regression loss + class loss. The optimizer can be either Adam or SGD; Learning rate scheduling can be either cosine annealing or Warmup combined with Cosine LR; Batch size and iteration: Batch size 16-32, iterations 200-500 epochs, adjusted based on validation set performance. Validation metrics include accuracy, recall, and F1 score.

[0118] The confidence threshold is adjusted during training to ensure high recall in the Open state.

[0119] The inference process of a trained model includes the following steps: (1) Acquiring images of the car door: Before the robot starts working, the robot's robotic arm or camera can be controlled to acquire images of the car door; (2) Preprocessing, including scaling the car door image to the input size specified by the model, and normalizing or standardizing the resized car door image. (3) Input the preprocessed car door image into the model, and the model outputs the car door bounding box and classification probability.

[0120] Emergency avoidance maneuvers refer to the sequence of safety actions a robot performs when it detects a safety hazard such as an open car door. This action typically involves rapidly retracting the two robotic arms towards the center of the robot body to minimize the robot's overall profile and avoid a collision with the vehicle.

[0121] In this embodiment, when the algorithm generates a quality inspection task trigger signal and confirms that a new vehicle has entered the quality inspection station in step S110, the algorithm does not immediately execute the subsequent follow-up data collection operation, but first starts the door safety inspection process.

[0122] The algorithm controls the robot to move to a preset door detection position. Specifically, the algorithm calls a predefined sequence of actions to adjust the robot chassis position and simultaneously control the dual robotic arms to move to preset joint angles and end-effector poses, ensuring that the vision sensors mounted at the ends of the robotic arms are directly facing the side door area of ​​the vehicle. This door detection position is predetermined during system deployment through teaching or calibration to ensure that the acquired door images have a suitable viewing angle and clarity.

[0123] Once the robot reaches the door detection position, the algorithm triggers the vision sensor to perform door image acquisition, obtaining an image of the vehicle's side door. The acquired image can be a single frame or a series of multiple frames, used for subsequent state recognition.

[0124] The algorithm inputs the acquired car door images into a pre-trained car door state detection model. This model, based on deep learning, can identify the car door's outline, gap features, and relative position to the car body from the image, thus determining whether the door is currently closed or open. The model output includes the door state category (e.g., "closed" or "open") and its confidence level.

[0125] The algorithm makes branch decisions based on the output of the door state detection model: (1) If the car door is closed: the algorithm determines that the current safety conditions are met and continues to execute the subsequent quality inspection process, that is, enter step S120 and the subsequent follow-up data collection operation.

[0126] (2) If the car door is open: The algorithm determines that there is a safety hazard and immediately suspends the current quality inspection task. To avoid the robotic arm colliding with the open car door during subsequent operations, the algorithm controls the robot to perform an emergency avoidance operation. This emergency avoidance operation typically includes: stopping the chassis movement and quickly retracting both robotic arms to the center of the robot body to minimize the overall outline of the robot. At the same time, the algorithm reports the abnormal information to the cloud server through the agent, including the detection result of the car door being open, the vehicle identification code, and the time of occurrence. After the abnormality is reported, the system can perform manual intervention or automatically wait according to the preset strategy until the safety hazard is eliminated before resuming operations.

[0127] Through the aforementioned door safety detection mechanism, this embodiment proactively identifies whether the door is closed before the robot performs quality inspection, enabling early prediction and avoidance of safety hazards. Compared to the passive safety approach in existing technologies where the robot directly extends into the inspection area and only triggers an emergency stop upon collision, this embodiment proactively identifies safety risks through visual perception, taking evasive action before the robotic arm contacts the vehicle, effectively preventing equipment damage and production line stoppages caused by open doors. Furthermore, this safety detection process is seamlessly integrated with the main quality inspection process, ensuring operational safety without affecting the normal inspection cycle, significantly improving the system's reliability and safety in complex industrial environments.

[0128] In some embodiments, before the robot performs the quality inspection data collection operation, the method further includes: the algorithm setting the shooting configuration information of the robot's vision sensor according to the vehicle model information; the shooting configuration information includes shooting parameters configured according to the vehicle's paint color type.

[0129] A vision sensor is an image acquisition device installed at the end of a robot's dual robotic arms. It is typically a high-definition industrial camera used to acquire vehicle images at quality inspection points. This sensor supports dynamic adjustment of parameters such as exposure time, gain, and aperture.

[0130] Shooting configuration information refers to the combination of parameters used by the vision sensor when performing image acquisition, including exposure time, gain (ISO), aperture size, white balance, etc. These parameters directly affect the quality of the acquired image, such as brightness, contrast, sharpness, and noise level.

[0131] In this embodiment, in step S120, the agent has sent the vehicle model information to the algorithm. The model information includes the current vehicle's paint color type, such as high-reflective white, dark charcoal black, matte paint, etc.

[0132] In step S130, before the algorithm controls the robot to perform the quality inspection data collection operation, it first determines the visual sensor shooting configuration information applicable to the current vehicle based on the paint color type in the vehicle model information.

[0133] Specifically, the algorithm has a pre-set shooting parameter configuration table, which records the mapping relationship between different car paint color types and visual sensor parameters. For example: (1) For highly reflective white car paint, due to its high surface reflectivity, it is easy to overexpose. Therefore, the configuration table should be set with a shorter exposure time, a lower gain value (such as ISO 100), and an appropriate reduction in aperture to prevent loss of details in the bright parts of the image.

[0134] (2) For dark carbon black car paint, due to its strong light absorption, the image is prone to being dark and the details are not clear. Therefore, the configuration table should be set with a longer exposure time and a higher gain value (such as ISO 400) to enhance the ability to extract details in the dark areas.

[0135] (3) For matte paint, its reflective properties are between the two. The corresponding medium exposure time and medium gain value should be set in the configuration table to balance image brightness and texture reproduction.

[0136] The parameters mentioned above are for illustrative purposes only. In actual applications, they can be precisely calibrated and adjusted according to the optical properties of different car paints, ambient lighting conditions, and testing requirements.

[0137] The algorithm queries a pre-defined shooting parameter configuration table based on the vehicle paint color type in the vehicle model information to obtain the corresponding shooting configuration information such as exposure time, gain, and aperture. Subsequently, the algorithm sends these configuration parameters to the vision sensor via a communication protocol, controlling the sensor to use these parameters for image acquisition in subsequent acquisition operations.

[0138] The configuration settings for the camera are completed before each quality inspection task, ensuring that the vision sensor can capture high-quality images for different colors and materials of car paint, providing reliable raw data for subsequent cloud-based defect detection.

[0139] Through the aforementioned adaptive shooting parameter mechanism, this embodiment achieves dynamic matching of visual sensor parameters to different car paint types. Compared to the traditional method of acquiring images with fixed parameters, this embodiment can automatically adjust parameters such as exposure and gain according to the color characteristics of the car paint, effectively avoiding overexposure problems of highly reflective car paint and underexposure problems of dark car paint, significantly improving the clarity and detail reproduction of the acquired images. This effect is particularly crucial for subsequent defect detection based on AI models. High-quality input images can improve the accuracy and recall rate of defect identification, reduce missed detections and false detections caused by image quality issues, thereby improving the overall reliability and detection accuracy of the quality inspection system.

[0140] It should be noted that, regarding the various steps included in the vehicle final assembly off-line quality inspection method provided in any of the above embodiments, unless explicitly stated herein, there is no strict order restriction on the execution of these steps; these steps can be executed in other orders. Moreover, at least some of these steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0141] This application also provides a robot for quality inspection of vehicles after final assembly, including: The chassis is used to drive the robot to move within the work area; Dual robotic arms, mounted on the chassis; Depth camera, used to acquire depth images in the direction of the conveyor belt; Vision sensors are installed at the ends of each robotic arm; Vehicle identification number acquisition device, used to acquire the vehicle identification number of a vehicle; The agent, deployed in the first computing unit, is communicatively connected to the vehicle identification code acquisition device; and On the algorithm side, it is deployed in the second computing unit and communicates with the agent, depth camera, vision sensor, chassis and dual robotic arms. The algorithm and the agent are used to execute the vehicle final assembly quality inspection method provided in any of the above embodiments.

[0142] The chassis serves as the robot's mobile base, typically employing differential drive or omnidirectional wheel structures to propel the robot autonomously within the assembly workshop's work area. The chassis integrates motor drivers, an odometer, and an inertial measurement unit, supporting precise speed control and position feedback.

[0143] The dual robotic arms are two multi-degree-of-freedom robotic arms mounted on the chassis. Each joint is driven by a servo motor and is used to move the vision sensor to the preset quality inspection work points. The dual robotic arm design allows the robot to inspect both sides of the vehicle simultaneously or alternately, improving work efficiency.

[0144] The depth camera is an RGB-D camera used to acquire depth and color images along the conveyor belt direction. This camera can perceive the three-dimensional spatial information of objects within its field of view, providing data support for new vehicle entry detection and front-end positioning. In some embodiments, the depth camera is mounted on top of the robot.

[0145] The vision sensors are high-definition industrial cameras mounted on the ends of the dual robotic arms, used to acquire high-definition images of vehicles at various quality inspection points. The vision sensors support dynamic adjustment of parameters such as exposure time, gain, and aperture to adapt to the imaging characteristics of different vehicle paints.

[0146] There are several ways to implement a vehicle identification number (VIN) acquisition device. For example, it can be a scanner. This scanner can be an RFID (Radio Frequency Identification) scanner, which is an ultra-high frequency RFID reader / writer installed on the robot itself. It transmits radio frequency signals to read the vehicle identification number (VIN) stored in a pre-set RFID electronic tag on the vehicle. RFID scanners have a long reading distance and strong anti-interference capabilities, enabling stable reading while the vehicle is moving. Another example is a camera, which captures the VIN code on the plate below the windshield or on the B-pillar and uses optical character recognition technology to identify the VIN. Yet another example is a device that can wirelessly communicate with the vehicle's onboard communication module via Wi-Fi, Bluetooth, UWB, or 5G networks to directly obtain the VIN.

[0147] The agent is a software module deployed in the primary computing unit (such as an x86-based industrial host) for non-real-time tasks such as data interaction with the cloud server, processing business logic, and managing hardware peripherals. The agent communicates with the cloud via Ethernet or a 5G module and with the algorithm via an internal bus.

[0148] The algorithm is a software module deployed in the second computing unit (such as the NVIDIA Jetson Orin embedded computing platform) to perform latency-sensitive tasks such as real-time perception, motion planning, and control. The algorithm directly drives hardware devices such as depth cameras, vision sensors, LiDAR, chassis, and dual robotic arms.

[0149] The first and second computing units are two independent computing devices within the robot, communicating via an internal network (such as Ethernet or PCIe). The first computing unit prioritizes stability and rich interface options, while the second computing unit prioritizes parallel computing capabilities and real-time performance. This heterogeneous computing architecture achieves physical isolation between business processing and real-time control.

[0150] The robot provided in this embodiment has the following hardware deployment and software module division: A depth camera is fixedly mounted on the robot, with its lens facing the conveyor belt, continuously acquiring depth image data of the conveyor belt. A vehicle identification code acquisition device is installed on the side or front of the robot, facing the direction in which vehicles pass, and is used to read the vehicle identification code when a vehicle enters the workstation.

[0151] The dual robotic arms are mounted on both sides of the chassis, with a vision sensor installed at the end of each arm. The robotic arms have multiple degrees of freedom, allowing for flexible adjustment of the spatial position and orientation of the vision sensors to adapt to the data acquisition needs of different vehicle models and different detection points.

[0152] In terms of software architecture, the robot is internally divided into two logical modules: the agent module and the algorithm module, which are deployed in the first and second computing units, respectively. The agent module and the algorithm module interact with each other via communication protocols such as ROS2 or MQTT.

[0153] The agent is responsible for communicating with the cloud server, including receiving conveyor belt speed data from the cloud (or obtaining conveyor belt speed data locally), uploading quality inspection data packets, etc.; it is also responsible for controlling the vehicle identification code acquisition device and reading data, obtaining the vehicle identification code, and querying the corresponding vehicle model information based on the vehicle identification code.

[0154] The algorithm is responsible for real-time perception and control, including: monitoring the status of the conveyor belt traffic through a depth camera to detect new vehicles entering the workstation; obtaining the initial position information of the vehicle relative to the robot; planning the quality inspection work points according to the vehicle model information sent by the agent; controlling the chassis to follow the vehicle's movement; controlling the dual robotic arms to move to each quality inspection work point and performing quality inspection data collection operations through vision sensors; and sending the collected quality inspection data to the agent after associating it with the vehicle identification code.

[0155] When performing quality inspection tasks, the algorithm and the agent work together to fully implement the vehicle final assembly quality inspection method described in any of the foregoing embodiments. For specific procedures, please refer to the descriptions of the foregoing method embodiments; they will not be repeated here.

[0156] The robot provided in this embodiment achieves physical isolation between business processing and real-time control by deploying the agent end and algorithm end on a heterogeneous computing unit. This effectively avoids interference from non-real-time tasks on real-time control tasks and ensures low-latency response for robot motion control and data acquisition. Simultaneously, the robot integrates multiple sensors, including a depth camera, dual-arm end-effector vision sensors, and a vehicle identification code acquisition device, forming a complete perception-decision-execution closed loop. It can autonomously complete the entire quality inspection process, from vehicle identification and location planning to data acquisition, in dynamic scenarios where vehicles are continuously moving. Compared to existing mobile robot solutions, this robot possesses stronger environmental perception capabilities, higher operational accuracy, and better system stability, meeting the flexible production needs of multi-model, high-cycle production in mixed-flow assembly modes.

[0157] In some embodiments, the robot further includes a lidar mounted on the chassis for building an environmental map and achieving real-time positioning.

[0158] LiDAR can be a single-line or multi-line LiDAR mounted on the chassis, used to scan the surrounding environment in real time, build an environmental map, and work with SLAM algorithms to achieve precise positioning and navigation of the robot in the final assembly workshop.

[0159] In some embodiments, the robot also includes a torque sensor embedded in the chassis sidewall and the joints of the dual robotic arms, for detecting the reverse impact force received by the robot; The algorithm is also used to trigger a hardware emergency stop when the reverse impact force detected by the torque sensor exceeds a preset threshold. The agent is also used to: respond to hardware emergency stop, forcibly shut down non-safety-related computing nodes, stop sensor transmission, and send a system offline signal to the cloud server; The robot also includes: electromagnetic brakes, which are installed at the joints of the dual robotic arms and the drive motors of the chassis, and are used to lock the robot in place when the hardware emergency stop is triggered, cut off the motor power, and fix the robot in the current position. And an audible and visual alarm device, used to issue an audible and visual alarm when the hardware emergency stop is triggered.

[0160] Torque sensors are high-frequency sensors embedded in the sidewalls of the robot chassis and at the joints of the dual robotic arms. They are used to detect reverse impact forces acting on the robot in real time. Torque sensors can detect abnormal force conditions such as collisions and compressions, and transmit the force data to the algorithm in real time.

[0161] The preset threshold is a pre-defined upper limit for the reverse impact force, such as 50 Newtons. When the detected impact force exceeds this threshold, the system determines that a collision or abnormal contact has occurred and triggers emergency stop protection. The preset threshold can be calibrated according to the robot's structural strength and the operating scenario.

[0162] Hardware emergency stop is the highest level of safety protection mechanism. When abnormal force is detected, the system immediately cuts off the motor power and locks the mechanical position, causing the robot to stop all movement instantly to prevent the accident from escalating or the equipment from being damaged.

[0163] Non-safety-related computing nodes refer to computational tasks in the robot software system that are unrelated to safety protection, such as image processing, data uploading, and user interface interaction. These nodes are forcibly shut down when an emergency stop is triggered to release system resources and avoid interfering with the safety response.

[0164] The electromagnetic brake is a braking device installed at each joint of the dual robotic arms and at the drive motor of the chassis. Under normal power supply, it is in the released state, and the motor can move freely. When an emergency stop is triggered, the brake is powered on and locked, using electromagnetic force to fix the motor rotor, so that the robot's joints and chassis stop instantly and remain in the current position.

[0165] The system offline signal is a status signal sent by the agent to the cloud server, indicating that the robot has exited service due to a malfunction or emergency stop and will no longer participate in quality inspection task scheduling. After receiving the signal, the cloud server marks the robot as "offline" or "faulty" and arranges for another robot to take over its work.

[0166] The audible and visual alarm device is an alarm system installed on the robot itself, consisting of an LED status light strip and a voice broadcast module. The LED light strip can switch between different colors and flashing modes, and the voice module can play preset fault codes or prompt voice messages.

[0167] In this embodiment, during robot operation, torque sensors embedded in the chassis sidewalls and the joints of the dual robotic arms continuously monitor the external reaction forces acting on the robot. When the robot's end effector accidentally collides with a vehicle, conveyor belt, or person, or when the robotic arm joints are subjected to external compression, the torque sensors detect the impact force data in real time and transmit it to the algorithm.

[0168] The algorithm continuously receives data from the torque sensor and compares the real-time detection value with a preset threshold (e.g., 50 Newtons). When the detected reverse impact force exceeds the preset threshold, the algorithm determines that an abnormal collision has occurred and immediately triggers the highest level of hardware emergency stop.

[0169] After a hardware emergency stop is triggered, the system executes a triple response: (1) Physical Lock-up. The algorithm sends a lock-up command to the electromagnetic brakes installed at each joint of the dual robotic arms and the drive motors of the chassis. The electromagnetic brakes are immediately energized, locking the motor rotors through electromagnetic force, cutting off the motor power output, and causing the robotic arms and chassis to stop moving instantly and be fixed in their current positions. This physical lock-up mechanism ensures that the robot will not continue to move due to inertia after a collision, avoiding secondary damage.

[0170] (2) Software Interruption. In response to the hardware emergency stop signal, the agent immediately and forcibly shuts down all non-safety-related computing nodes, including image processing tasks, data upload threads, and user interface interactions. At the same time, the agent stops sending transmission commands to sensors such as vision sensors and LiDAR, reducing system power consumption and avoiding interference signals in abnormal conditions. The agent also sends a system offline signal to the cloud server, reporting that the robot has exited service due to the emergency stop event.

[0171] (3) Audible and visual alarm. The agent terminal simultaneously controls the audible and visual alarm device to start: the LED status light strip switches to red high-frequency flashing mode to conspicuously remind on-site personnel that the robot is in an emergency stop state; the voice module plays preset fault codes (such as "E001 - Collision Emergency Stop") in a loop to facilitate operators to quickly identify the fault type.

[0172] Through the aforementioned hierarchical emergency stop protection mechanism, this embodiment achieves a complete safety response chain from collision detection to physical locking, software interruption, and audible and visual alarms. Compared to the passive protection methods in existing technologies that rely on external safety fences or manual intervention after a collision, this embodiment achieves active collision detection and millisecond-level response through an embedded torque sensor. It can cut off power and lock the position instantly when the impact force exceeds the threshold, effectively preventing the accident from escalating and equipment damage. Simultaneously, through software interruption at the agent end and offline reporting to the cloud, the system can promptly remove the faulty robot from the scheduling queue, avoiding subsequent incorrect task allocation and providing reliable safety assurance for multi-robot collaborative operations. This mechanism significantly improves the operational safety and system robustness of robots in complex industrial environments.

[0173] In some embodiments, the robot also includes an interactive interface disposed on the robot body for displaying the cause of the emergency stop and receiving manual input of a recovery command; The agent is also used to: display the cause of the emergency stop through the interactive interface after the hardware emergency stop is triggered; execute the hot restart process in response to the recovery command entered through the interactive interface to restart non-security-related computing nodes; and re-request work instructions from the cloud server after the hot restart is completed. The algorithm is also used to: after a hot restart, control the robot to perform slow obstacle avoidance actions and return to the preset trigger starting point position to prepare for the next quality inspection task.

[0174] The interactive interface is a touch screen or display panel with physical buttons located on the robot body, used for human-machine interaction with the operator. The interactive interface can display the robot's operating status, fault information, emergency stop reasons, etc., and receive recovery commands input by the operator (such as clicking the "Emergency Stop Recovery" virtual button).

[0175] The emergency stop reason refers to the specific cause that triggers the hardware emergency stop, such as "robotic arm collision," "chassis side impact," or "torque over-limit." The emergency stop reason is determined by the algorithm based on torque sensor data and displayed on the interactive interface through the agent, making it easier for operators to understand the fault type and take targeted measures.

[0176] A recovery command is a confirmation command entered by the operator through the interactive interface after clearing obstacles from the site, indicating that the site is safe and work can resume. The recovery command is typically obtained by clicking the "Emergency Stop Resume" virtual button on the interactive interface.

[0177] A warm reboot is a process that restarts only non-security-related compute nodes without restarting the entire system image. Warm reboots are faster than cold boots (powering on or restarting the operating system) and do not affect already loaded system configurations or calibrated sensor parameters. Warm reboots are achieved by executing the `Node_Reload` command, reloading only software nodes and not involving hardware initialization.

[0178] Non-safety-related computing nodes refer to computational tasks in the robot software system that are unrelated to safety protection, such as image processing nodes, data upload nodes, and user interface interaction nodes. These nodes are forcibly shut down when an emergency stop is triggered; during recovery, they are restarted via a warm reboot, restoring the robot to a workable state. Safety-related nodes (such as torque monitoring and emergency stop control) remain operational during the emergency stop to ensure the safety of the recovery process.

[0179] Slow obstacle avoidance refers to a sequence of low-speed movements performed by the robot after a warm restart. The robot moves at a speed lower than its normal operating speed, while using LiDAR and vision sensors to perceive the surrounding environment in real time, avoid obstacles, and safely return to the preset position.

[0180] The trigger start point refers to the initial position of the robot when performing a quality inspection task, which usually corresponds to the robot's preset stopping point when the vehicle enters the workstation. This position is determined through teaching or calibration during system deployment and serves as the starting reference for each quality inspection task. After returning to the trigger start point, the robot can prepare to perform the quality inspection task for the next vehicle.

[0181] In this embodiment, when the robot triggers a hardware emergency stop due to a collision or other reasons, the robot enters an emergency stop locked state. At this time, the agent displays the emergency stop cause determined by the algorithm (such as "left robotic arm joint collision, impact force 65N") through the interactive interface, and simultaneously alerts the on-site operator with an audible and visual alarm device.

[0182] Upon hearing the audible and visual alarm, the operator first observes the situation and investigates the cause of the emergency stop, such as removing the collision object, adjusting the vehicle's position, or clearing obstacles. Once the safety hazard has been confirmed, the operator clicks the "Emergency Stop Resume" virtual button on the interactive interface and enters the recovery command.

[0183] The agent responds to the recovery command and initiates a warm reboot process. The agent executes the Node_Reload command, restarting only non-security-related compute nodes (such as image processing, data uploading, and user interface nodes), without performing a cold boot of the operating system. The warm reboot process typically completes within seconds, significantly reducing recovery time.

[0184] After a hot reboot is complete, the agent sends a request to the cloud server to reapply for a work instruction. Once the cloud server confirms that the robot has returned to online status, it is reinstated into the scheduling resource pool.

[0185] Simultaneously, the algorithm initiates a reset and origin-finding process. The algorithm controls the robot chassis and dual robotic arms to perform slow obstacle avoidance maneuvers, moving at a rate lower than normal operating speed. During this movement, the algorithm uses LiDAR to scan the surrounding environment in real time, combining this with a pre-set environmental map to perform dynamic obstacle avoidance planning, ensuring the robot safely traverses the workshop aisles. The algorithm then controls the robot to gradually return to the pre-set trigger starting point (Home position), which corresponds to the robot's initial stopping point when the vehicle enters the workstation.

[0186] Once the robot reaches the trigger point, the algorithm updates the robot's status to "ready" and reports the ready status to the cloud server through the agent. At this point, the robot has completed the full self-healing recovery process and can participate normally in the scheduling and execution of subsequent quality inspection tasks.

[0187] Through the aforementioned interactive recovery and logical self-healing mechanisms, this embodiment achieves rapid recovery after an emergency stop, significantly shortening production downtime. Compared to the cumbersome process of manually restarting, recalibrating, and redeploying the system after an emergency stop in existing technologies, this embodiment uses hot restart technology to only restart non-safety-related nodes, avoiding the long wait time of a cold start; through slow obstacle avoidance and automatic reset, the robot can autonomously return to the preset starting point without manual guidance; and by automatically re-requesting work instructions from the cloud, seamless integration after recovery is achieved. This mechanism significantly reduces the degree of manual intervention and fault recovery time, improving the availability and operational continuity of the robot on high-paced production lines. Simultaneously, displaying the cause of the emergency stop through the interactive interface provides operators with clear fault diagnosis information, facilitating rapid problem location and precise handling, further shortening the average repair time.

[0188] This application also provides a vehicle final assembly line quality inspection system; please refer to [link / reference]. Figure 2 The system includes: A cloud server and at least one robot provided in the above embodiments for quality inspection of vehicle final assembly line; The cloud server is used to: receive quality inspection data uploaded by the robot's agent, perform defect detection based on the quality inspection data, and generate defect detection results.

[0189] Cloud servers are server clusters deployed in remote data centers or local factory data centers, equipped with high-performance GPU computing resources to run large-scale AI defect detection models. The cloud servers interact with the agents of each robot via industrial Ethernet or 5G networks.

[0190] Quality inspection data consists of image or sensor data collected by the robot's algorithm at various quality inspection points, along with metadata related to the data collection (such as collection time, robot status, etc.). The quality inspection data is packaged and encrypted by the robot's agent before being uploaded to the cloud server.

[0191] The defect detection results are output by a cloud server based on a pre-trained AI model analyzing quality inspection data. These results include the defect type (such as scratches, dents, and paint defects), defect location, severity, and whether the defect is acceptable. The detection results are then stored in a database after being linked to the vehicle identification number (VIN).

[0192] The vehicle final assembly quality inspection system provided in this embodiment consists of a cloud server and at least one quality inspection robot working together to complete the quality inspection operation.

[0193] When a vehicle enters the quality inspection station, the robot performs quality inspection operations according to the process described in the aforementioned method embodiment. Specifically, the robot's algorithm detects the entry of a new vehicle using a depth camera, while the agent obtains the vehicle identification code and queries the vehicle model information using a vehicle identification code acquisition device, such as an RFID scanner. Based on the vehicle model information and initial position information, the algorithm plans the quality inspection operation points and controls the robot to sequentially perform quality inspection data collection operations while following the vehicle.

[0194] After data collection is complete, the algorithm packages the quality inspection data along with metadata such as vehicle identification number, collection time, and robot status, and sends it to the agent. The agent verifies the integrity of the data packet and then uploads it to the cloud server via HTTPS or a dedicated API interface.

[0195] After receiving the data packet, the cloud server parses out the quality inspection data and vehicle identification number (VIN). The cloud server then calls a pre-trained and deployed AI defect detection model to analyze the quality inspection data. The AI ​​model uses a deep convolutional neural network architecture and can identify scratches, dents, paint defects on the vehicle body surface, as well as the assembly status and model markings of parts. For detection results with low confidence levels output by the AI ​​model, the cloud server can automatically push them to a human quality inspection station for remote secondary verification, where quality inspectors manually confirm any questionable results.

[0196] The cloud server associates the final defect detection results with the vehicle identification number and stores them in the production quality database, forming a complete quality traceability record. Simultaneously, the cloud server can feed the detection results back to the production execution system or a visual dashboard in real time according to preset rules, allowing production managers to monitor the quality status in real time.

[0197] The vehicle final assembly line quality inspection system provided in this embodiment achieves global optimization and data closure of quality inspection operations through layered collaboration between cloud servers and robots. The cloud server centrally processes AI defect detection tasks with high computing power requirements, avoiding the increased costs and power consumption issues associated with deploying high-performance computing hardware locally on the robots. Simultaneously, the cloud server uniformly manages the speed data distribution and inspection result archiving of multiple robots, laying the foundation for subsequent multi-robot collaborative scheduling and data analysis. Compared to existing single-robot quality inspection solutions, this system possesses stronger computing power, better data traceability, and higher system scalability, meeting the comprehensive needs of large-scale production lines for quality inspection efficiency, detection accuracy, and quality traceability.

[0198] In some embodiments, the cloud server is also used to: send real-time speed data of the conveyor belt to the robot's agent.

[0199] During system operation, the cloud server continuously collects real-time speed data of the conveyor belt from the production line's programmable logic controller (PLC) and sends the speed data to the agent terminals of each robot via MQTT or WebSocket protocol at a preset frequency (e.g., 10Hz). After receiving the speed data, the agent terminals forward it to the algorithm terminal for integration and accumulation to calculate the real-time displacement of the vehicle.

[0200] In some embodiments, the cloud server is also used to: plan a path for the robot to a charging station when the battery level of any robot is lower than a preset battery threshold, and assign other robots that meet preset conditions to take over the work of the robot.

[0201] The preset battery threshold is a pre-defined lower limit for battery power, such as 20% or 30%. When the robot's battery power falls below this threshold, the system determines that the robot needs to go to the charger to avoid task interruption due to battery depletion during operation.

[0202] The preset conditions include, but are not limited to: the robot is in standby mode, the robot's current task can be safely interrupted, the robot is about to complete its current task, the robot is closest to the current workstation, and the robot has the necessary functional configuration to perform the current quality inspection task. The cloud server selects the optimal replacement from robots that meet the preset conditions based on a real-time scheduling strategy to ensure the continuity of quality inspection operations and the efficient use of system resources.

[0203] The charging stations are deployed in the automated charging system within the final assembly workshop's work area. Robots can autonomously navigate to the charging stations to recharge. The location coordinates of the charging stations are pre-calibrated and stored on a cloud server.

[0204] Standby mode refers to a robot that has completed charging or is idle, not performing quality inspection tasks, and has sufficient power to be scheduled to take over the work of other robots.

[0205] In some embodiments, during system operation, each robot periodically (e.g., every 30 seconds) reports its operational status information to the cloud server via an agent. The operational status information includes: current battery percentage, current working status (e.g., "performing a task", "standby", "charging", "fault"), the identifier of the currently executing quality inspection task, and the estimated task completion time.

[0206] The cloud server receives and stores the operating status information of each robot, monitoring the battery status and workload of all robots in real time. Battery status refers to the current remaining battery power of the robot, usually expressed as a percentage. The robot periodically reports its battery status to the cloud server through an agent. Workload refers to the robot's current workload, including whether it is currently performing a quality inspection task, the task queue length, and the estimated completion time. Workload is used to determine whether the robot can be scheduled to perform a new task.

[0207] The cloud server continuously monitors whether the battery level of each robot is below a preset threshold (e.g., 20%). When a robot's battery level is detected to be below this threshold, the cloud server initiates the charging scheduling process: First, the cloud server retrieves the coordinates of charging stations near the robot's workstation from the map database. Then, it uses a path planning algorithm to plan a collision-free path from the robot's current location to the target charging station. The path planning considers both static obstacles (such as equipment and columns) and dynamic obstacles (such as other mobile robots and personnel) within the workshop to ensure that the robot can reach the charging station safely and efficiently.

[0208] Secondly, the cloud server queries all robots that are currently in standby mode and have sufficient power, and selects one from them as the replacement robot. The selection strategy can comprehensively consider factors such as the distance between the replacement robot and the current workstation, its power reserves, and whether there are other tasks waiting to be performed.

[0209] The cloud server sends a task takeover command to the replacement robot. The command includes vehicle information, task parameters, and workstation location for the quality inspection task to be performed. After receiving the command, the replacement robot autonomously navigates to the target workstation and takes over the subsequent quality inspection task.

[0210] Meanwhile, the cloud server sends a charging command to the robot with low battery, which includes a planned charging path. After receiving the command, the robot, either by completing its current quality inspection task (if in progress) or immediately terminating its standby state (if idle), autonomously moves to the charging station according to the planned path to charge. During charging, the robot continues to report its battery status to the cloud. When the battery level recovers to a preset charging completion threshold (e.g., 90%), the cloud server updates its status to "standby" and reinstates it to the schedulable resource pool.

[0211] After the robot successfully takes over the task, the cloud server updates the task allocation record to ensure the continuity of quality inspection operations and the accuracy of data correlation.

[0212] Through the aforementioned multi-machine collaboration and charging management mechanism, this embodiment achieves autonomous charging and seamless task handover for the quality inspection robot. When the robot's battery is low, the system can automatically plan a charging path and assign an idle robot to take over the work, without human intervention. This enables the quality inspection station to operate continuously 24 / 7, avoiding station downtime and production line interruptions caused by robot charging. Compared to the passive approach in existing technologies where robots leave their posts to charge after automatically determining low battery levels, resulting in station vacancies, this embodiment significantly improves system availability and operational continuity through unified cloud scheduling and task handover, making it particularly suitable for high-cycle, large-scale automotive assembly line scenarios.

[0213] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0214] Those skilled in the art will understand that all or part of the processes in the above method embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

Claims

1. A method for quality inspection of vehicles after final assembly, characterized in that, The method is applied to a robot, which includes an agent and an algorithm; the method includes: The algorithm detects whether a new vehicle enters the quality inspection station based on the conveyor belt monitoring data from the depth camera. If so, it triggers a quality inspection task and obtains the initial position information of the vehicle relative to the robot. In response to the trigger signal of the quality inspection task, the agent obtains the vehicle identification code of the vehicle, determines the vehicle model information based on the vehicle identification code, and sends the vehicle model information to the algorithm. The algorithm determines multiple quality inspection work points during the vehicle's movement along the conveyor belt based on the vehicle model information and the initial position information, and controls the robot to sequentially perform quality inspection data collection operations at the multiple quality inspection work points while following the vehicle to obtain quality inspection data. The quality inspection data is uploaded to the cloud server via the agent for defect detection.

2. The method according to claim 1, characterized in that, The algorithm detects whether a new vehicle is entering the quality inspection station based on conveyor belt monitoring data from a depth camera, including: The algorithm identifies the changes in the number of vehicles within the field of view of the depth camera based on the depth image of the conveyor belt direction generated by the depth camera, and generates a traffic flow status sequence. The traffic flow status sequence is compared with a predefined pattern set, which includes various patterns representing changes in the number of new vehicles entering the quality inspection station. When the traffic flow state sequence matches any of the patterns in the predefined pattern set, it is determined that a new vehicle has entered the quality inspection station.

3. The method according to claim 1, characterized in that, The algorithm determines multiple quality inspection points during the vehicle's movement along the conveyor belt based on the vehicle model information and the initial position information, including: The vehicle body length is obtained based on the vehicle model information; The initial distance between the front of the vehicle and the robot is determined based on the initial position information; Based on the vehicle body length and the initial distance, the target distance values ​​corresponding to each of the multiple quality inspection work points are determined, and the multiple quality inspection work points correspond to the front, middle and rear sections of the vehicle, respectively.

4. The method according to claim 3, characterized in that, The step of determining the target distance values ​​corresponding to each of the multiple quality inspection work points based on the vehicle body length and the initial distance includes: Based on the vehicle body length and the initial distance, determine the theoretical distance values ​​corresponding to each of the multiple quality inspection operation points at the front, middle and rear sections of the vehicle; Based on the weighted average of the system delay time and the recent speed of the conveyor belt, the theoretical distance value corresponding to each of the quality inspection operation points is compensated to obtain the target distance value corresponding to each of the quality inspection operation points.

5. The method according to claim 3 or 4, characterized in that, The process of controlling the robot to sequentially perform quality inspection data collection operations at multiple quality inspection work points while following the vehicle includes: The algorithm determines the target quality inspection operation point from the plurality of quality inspection operation points to perform the quality inspection data collection operation. The robot is controlled to move a distance corresponding to the target quality inspection point; the moving distance corresponding to the target quality inspection point is less than the target distance corresponding to the target quality inspection point. Based on the real-time conveyor belt speed issued by the cloud server, the real-time displacement of the vehicle as it moves along the conveyor belt from the moment the quality inspection task is triggered is calculated by accumulating the data using an integral algorithm. When the real-time moving displacement reaches the target distance value corresponding to the target quality inspection work point, the robot is controlled to perform quality inspection data collection operation; Determine whether the quality inspection data collection operation has been completed at all the aforementioned quality inspection operation points. If not, return to the step of determining the target quality inspection operation point from the multiple quality inspection operation points to perform the quality inspection data collection operation.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: The algorithm controls the robot to move to the door detection position to perform door image acquisition; The system uses a door state detection model to identify the collected door images in order to determine whether the door is closed. If the car door is open, the quality inspection task is stopped, the robot is controlled to perform an emergency avoidance operation and report the abnormality.

7. The method according to any one of claims 1-4, characterized in that, Before the robot performs the quality inspection data collection operation, the method further includes: The algorithm sets the shooting configuration information of the robot's vision sensor according to the vehicle model information; the shooting configuration information includes shooting parameters configured according to the vehicle's paint color type.

8. A robot for quality inspection of vehicles after final assembly, characterized in that, include: A chassis for driving the robot to move within the work area; Dual robotic arms are mounted on the chassis; Depth camera, used to acquire depth images in the direction of the conveyor belt; Vision sensors are respectively installed at the end of each of the robotic arms; Vehicle identification number acquisition device, used to acquire the vehicle identification number of a vehicle; The agent is deployed in the first computing unit and is communicatively connected to the vehicle identification code acquisition device; as well as The algorithm is deployed in the second computing unit and is communicatively connected to the agent, the depth camera, the vision sensor, the chassis, and the dual robotic arms. The algorithm and the proxy are used to execute the method described in any one of claims 1 to 7.

9. A vehicle final assembly line quality inspection system, characterized in that, include: Cloud server; as well as At least one robot as described in claim 8; The cloud server is used for: The robot receives quality inspection data uploaded by its agent, performs defect detection based on the quality inspection data, and generates defect detection results.

10. The system according to claim 9, characterized in that, The cloud server is also used for: When the battery level of any of the robots falls below a preset battery threshold, a path is planned for the robot to reach a charging station, and another robot that meets the preset conditions is assigned to take over the work.

Images