Vehicle Quality Control System

Abstract

The adapter for securing a guide needle to a surgical instrument includes a body configured to be secured to the surgical instrument, first and second coupling members secured to the body, and first and second gripping members pivotally coupled to the body at the first and second coupling members, respectively. The first and second gripping members are configured to grasp a structure located at the proximal end of the guide needle to securely couple the guide needle to the surgical instrument. A related surgical system includes a surgical instrument having a probe, a guide needle configured to receive the probe, and an adapter for securing the guide needle to the surgical instrument. Related methods include methods of using the adapter and methods of performing a surgical procedure.

Description

[Technical Field]

[0001] The present invention relates to the field of vehicle quality control, such as quality control in vehicle production facilities. [Background technology]

[0002] Vehicle production facilities typically include a production line where vehicles are assembled through individual production stages. Vehicle production begins in a first stage, and the final vehicle assembly from the first production stage is sent to subsequent production stages, where the vehicle assembly is modified, for example, by adding components or processing or modifying existing components. Vehicle assemblies progress through production stages until they reach a completion stage. A production facility may have multiple completion stages, such as a stage before the paint shop when the body is complete, when body assembly and painting are complete, near the end of the line where the completed vehicle undergoes final testing, and before leaving the factory. A typical vehicle production line may include, for example, 500-600 production stages and 1-10 completion stages.

[0003] It is also known to provide a quality inspection stage after the completion stage. In one example, a quality control operator has a limited amount of time to visually inspect the completed vehicle to identify defects that must be corrected before the completed vehicle leaves the production facility. To maintain production volume, the quality control operator has, for example, 60 seconds or so to identify defects such as dents, dents, warps, scratches, chips, paint stains, and / or check whether the vehicle meets required specifications, where defects may include vehicles with incorrect wheels, badges, or other features.

[0004] If defects are found, the quality control operator enters the details into the production control system and when the vehicle reaches the repair station, a technician is instructed to correct the defects or deviations before the finished vehicle moves to the next stage of production or before the vehicle leaves the production facility.

[0005] Generally, the earlier a defect is found on the production line, the cheaper it is to resolve. Even relatively small deviations can be more costly to correct if a vehicle leaves the production line with a defect, and if a defect is identified late, the problem may not be investigated for weeks or months, resulting in a large number of vehicles having the same defect by the time the problem is addressed.

[0006] Similarly, transportation from the production line to the first owner or dealer may involve multiple logistics companies transporting the vehicle and may involve additional operations, such as fitting additional options to the vehicle, thus introducing defects along the logistics supply chain.

[0007] It is known to provide camera-based imaging systems during quality control to facilitate the rapid acquisition of images of completed vehicles. The images can be used to assess liability for damage or detect defects. For example, the images can be evaluated by a remote quality control operator, by traditional computer vision algorithms, or by machine vision algorithms to assess whether vehicle performance meets specification requirements or whether defects exist.

[0008] The present inventors have recognized that known quality control imaging systems in vehicle production facilities can be improved. Summary of the Invention

[0009] According to a first aspect of the present invention, there is provided a computer-implemented method for assessing quality of vehicles produced on a vehicle production line, the method comprising: the production line having a plurality of individual vehicle production stages arranged to define a production run between an initial production stage and a final production stage; The method comprises: identifying a vehicle assembly at a first production imaging station, acquiring a first set of one or more production images of the vehicle assembly after processing at a first production stage, and storing the first set of production images; Optionally, identifying the vehicle assembly at a second production imaging station, obtaining a second set of one or more production images of the vehicle assembly after processing at a second (e.g., later) production stage separate from the first production stage, and storing the second set of production images; identifying the vehicle assembly at a back-end imaging station and acquiring one or more evaluation images of the vehicle assembly after the back-end production processing; running a defect detection algorithm using the evaluation image to identify defects on the vehicle; acquiring the production image in response to identifying a defect on the vehicle by the defect detection algorithm; A computer-implemented method is provided, comprising:

[0010] Thus, a quality control method according to a first aspect of the present invention acquires production images of a vehicle assembly between multiple production stages. The production images can be, for example, images of one or more components of the vehicle assembly as it is assembled. Thus, as the vehicle assembly is processed by one or more production stages between acquiring the first and second production images, the production images define a visual record of the vehicle being assembled, i.e., the second set of production images shows the vehicle assembly in a more complete state than the first set of production images. Once the vehicle assembly is processed in the final production stage, evaluation images are acquired and processed by a defect detection algorithm to identify whether defects exist on the vehicle. Thus, the final stage imaging station is a quality control imaging stage that looks for defects such as scratches, scuffs, dirt, dents, crumbs, chips, paint contamination, alignment or distortion issues, and specification deviations. If defects are detected by the defect detection algorithm, production images are acquired. Because the production images show the vehicle assembly at a discrete point or points along the production process, the presence or absence of a defect in the production images can be used to identify the production stage at which the defect first occurred, allowing the source of a quality control problem in the vehicle production line to be determined as quickly as possible, potentially in real time. It is understood that the initial and final production stages may form part of a larger production process, i.e., they need not be the absolute start and end of the production line.

[0011] Preferably, the method includes acquiring a plurality of evaluation images.

[0012] In one or more, and in any of several embodiments, the set of production images each includes a smaller number of images than the set of evaluation images, and in one or more, and in any of several embodiments, the set of production images each includes images of lower resolution than the set of evaluation images.

[0013] The disclosed methods may include taking additional or alternative measurements and data beyond just images, for example, using 3D laser scanners, structured light scanners, stereoscopic or photogrammetry solutions to take 2D or 3D measurements and comparing these at different stages of the production process. These techniques can be used to identify defect locations. The imaging station may be used to locate defects in the images on a 2D or 3D model of the vehicle, to generate a butterfly diagram showing the defects, or to create a heat map over time. Thus, as used herein, the terms imaging station and image are not limited to the acquisition of visual information only.

[0014] The disclosed method may further include acquiring one or more additional sets of production images of one or more vehicle assemblies after processing at additional individual production stages and storing the additional sets of production images. A larger number of sets of production images provides greater resolution for quickly determining the source of a problem on the vehicle production line. Preferably, the disclosed method includes acquiring at least two sets of production images along the production run, and in some embodiments, the disclosed method includes acquiring at least 3, 5, 10, 20, 50, and in some cases more than 1000 sets of images.

[0015] After acquiring production images in response to a defect being identified on a vehicle by a defect detection algorithm, the disclosed method may include generating a report that includes at least one evaluation image showing the defect and the production image. Such a report can help a quality control operator recognize the occurrence of the defect and then quickly and easily review the production images to identify the production image in which the defect first occurred.

[0016] The disclosed method may include, after acquiring production images in response to the defect detection algorithm identifying a defect on the vehicle, running a defect location algorithm to search the production images for the defect identified in the evaluation images to identify a first set of production images that exhibit the defect. The disclosed method may also include generating a report identifying a production stage prior to the first occurrence of the defect in the set of production images. Such an embodiment may allow for earlier identification of the location of the production stage at which the defect occurs.

[0017] Each production image, or each set of production images, can be associated with the production stage at which the vehicle assembly was last processed when the image was taken, which allows a quality control operator to more easily associate an image with a specific location in the production process.

[0018] The disclosed method may include acquiring production images only in response to a defect on a vehicle being identified by a defect detection algorithm, which may improve network efficiency because production images are only transmitted when necessary to corroborate the identified defect.

[0019] The disclosed method may include deleting production images if the defect detection algorithm does not identify any defects on the vehicle, which may be particularly advantageous for vehicle imaging systems in terms of image file size, as it may reduce storage requirements.

[0020] The step of identifying the vehicle may include obtaining and storing an identifier for the vehicle assembly, such as a VIN, with each set of production images and evaluation images.

[0021] The step of identifying the vehicle may include obtaining production status information from a production line encoder / management system and / or a plant management system to obtain data indicative of the status of the vehicle assembly and / or production line. The disclosed method may also include comparing one or more defects with other data sets to identify trends, such as whether the frequency of defect occurrence is increasing or decreasing, whether changes are associated with other events such as a new process being implemented, whether more or less defects occur during different shifts or when particular workers are present, whether some vehicle types or vehicle specifications are more or less likely to experience defects, whether defects are more prevalent in particular areas of the vehicle, or whether defects are more prevalent in particular areas of the vehicle to identify defect location trends. A step of identifying whether the event is more or less likely to occur may be included.

[0022] At least one production imaging station can be movably mounted, and the method of the present disclosure can include an initial step of moving the production imaging station from a first position during a first pair of production stages to a second position during a second pair of production stages different from the first pair. Thus, the production imaging station can preferably be repositioned as needed, for example, so that it can be moved to a production stage suspected of generating defects.

[0023] The resulting process images show defects in different camera frames and at different stages of the production process and can be used to improve the training of defect detection algorithms. Defects only need to be identified and confirmed once, and using that localization data, the defects can be identified in multiple frames and multiple locations and automatically labeled, bounding boxes can be created, and the data can be used to train machine learning models for different stages of the process.

[0024] Where the physical size of the production and delivery process makes it impractical to run the system over a local network, a cloud storage and processing approach may be provided, creating a hybrid approach. While this increases the number of images that need to be transferred, the first aspect of the invention reduces long-term storage costs by reducing the need to store multiple images from early stages of the process that do not indicate defects or deviations. Similarly, there are benefits to storing results and final stage images in cloud storage to enable remote assessment of plant and shipment quality, as well as to allow different teams to evaluate data, compare trends at different plants, etc.

[0025] According to a second aspect of the present invention, there is provided a vehicle production facility, comprising: a vehicle production line having a plurality of individual vehicle production stages arranged to define a production run between an initial production stage and a final production stage; 1. A quality control imaging system, comprising: one or more production imaging stations located along the production process between the initial production stage and the final production stage, each production imaging station configured to capture a set of one or more production images of a vehicle assembly processed at a production stage; a back-end imaging station positioned to acquire a set of multiple evaluation images of the vehicle assembly processed at the back-end production stage; an identification system arranged to identify a vehicle assembly entering each imaging station; a first controller communicatively connected to the back-end imaging station, the first controller configured to receive the evaluation images acquired by the back-end imaging station and to execute a defect detection algorithm using the evaluation images to identify defects in the vehicle; a second controller communicatively connected to the production imaging station, the second controller configured to store the production images; a third controller communicatively coupled to the first controller and the second controller, the third controller configured to acquire the production image in response to the defect detection algorithm identifying a defect on the vehicle; a quality control imaging system having A vehicle production facility is provided, comprising:

[0026] The back-end imaging station has a modular configuration and includes a first surface defect detection module. The surface defect detection module may include a first surface detection module, a second surface defect detection module, and a dent detection module. Each surface defect detection module may include a frame, an imaging background surface mounted on the frame, and one or more first surface detection cameras mounted on the frame and oriented to view the imaging background surface in reflection from the vehicle assembly as the vehicle assembly moves through the module. The imaging background surface of the first surface detection module may be relatively bright or dark compared to the imaging background surface of the second surface detection module. In one example, the background surface of the first surface detection module may be a white, illuminated surface, while the imaging background surface of the second surface detection module may be gray or black and may be a non-reflective and / or non-illuminated surface. The frames of the first surface detection module, the second surface detection module, and the dent detection module may include standardized mounting points, allowing each module to quickly and easily connect to any other module at any stage. Each module may include a module controller configured to function as one of the first controller or second controller, making each module essentially "plug and play."

[0027] Each module can include an arch that extends around the sides and top of the module to define a path for the light-controlled vehicle assembly through the module. When two or more modules are attached together, their respective covers connect to form a continuous light-controlled vehicle assembly path through the imaging station. This embodiment allows for controlled light imaging the sides and top of the vehicle assembly.

[0028] For example, if only one side of a vehicle assembly needs to be imaged by a particular production imaging station, the production imaging station can include a partial arch that can extend over exactly one-half, one-third, or one-quarter of the full arch defined by the surface and dimple detection module of the final stage imaging station. According to this embodiment, the partial arch module can include covers that extend around both sides and the top of the module and define a path for the light-controlled vehicle assembly through the module.

[0029] Each production imaging station can be comprised of one or more modules present in the back-end imaging station. Thus, the production imaging stations can be simpler in terms of mechanical footprint / size and / or configuration than the back-end imaging stations, thereby reducing the overall size and cost of the quality control system.

[0030] Each module can be movably positioned, allowing imaging station modules to be easily moved from one location to another within the production line, enabling relocation of production imaging, such as repositioning the production imaging station as desired to move to locations where defects are expected. Data regarding defect trends can identify locations within the plant and on the vehicle where these defects occur. This information can then be used to move the imaging station to areas of interest within the plant. By identifying areas on the vehicle where problems occur, a complete arch is not required, and local cameras and lighting solutions can be easily positioned to check specific areas of the vehicle, such as the lower door jamb.

[0031] The first controller can be connected to a back-end imaging station via a wired connection. In use, the back-end imaging station can rapidly acquire hundreds to tens of thousands of high-resolution images of a vehicle assembly that are processed by defect detection algorithms. This allows for large amounts of data transfer without utilizing the vehicle production facility's wireless network bandwidth.

[0032] The optional features of the first aspect may be applied to the second aspect in an analogous manner.

[0033] According to a third aspect of the present invention, there is provided a quality control imaging system for use in a vehicle production facility according to the second aspect, comprising: one or more production imaging stations located along the production process between the initial production stage and the final production stage, each production imaging station configured to capture a set of one or more production images of a vehicle assembly processed at a production stage; a back-end imaging station positioned to acquire a set of evaluation images of the vehicle assembly processed at said back-end production stage; an identification system arranged to identify a vehicle assembly entering each imaging station; a first controller communicatively connected to the back-end imaging station, the first controller configured to receive the evaluation images acquired by the back-end imaging station and to execute a defect detection algorithm using the evaluation images to identify defects in the vehicle; a second controller communicatively connected to the production imaging station, the second controller configured to store the production images; a third controller communicatively coupled to the first controller and the second controller, the third controller configured to acquire the production image in response to the defect detection algorithm identifying a defect on the vehicle; A quality control imaging system is provided, comprising:

[0034] The optional features of the first and second aspects may be applied to the third aspect in an analogous manner.

[0035] According to a fourth aspect of the present invention, there is provided a computer-implemented method for assessing the quality of a vehicle as it passes through a plurality of distinct journey stages between an initial stage and a final stage, comprising the steps of: identifying a vehicle assembly at a first journey imaging station, acquiring a first set of one or more journey images of said vehicle after a first operational stage of processing, and storing said first set of journey images; identifying the vehicle at a final stage imaging station and acquiring one or more evaluation images of the vehicle after processing at the final journey stage; running a defect detection algorithm using the evaluation image to identify defects on the vehicle; acquiring the journey image in response to the defect detection algorithm identifying a defect on the vehicle; A computer-implemented method is provided, comprising:

[0036] Thus, the technical concept of the invention of the first aspect can be broadly applied to vehicle production and shipping processes, where the vehicle production journey stage can include the production stage, the shipping stage, or both. The same is true for the second and third aspects. In both stages, the process can consist of multiple operations to assemble the vehicle, address deviations from specifications, repair defects, or inspect the vehicle. Also, the logistics chain of the completed vehicle has similar processes to the production plant, only distributed. Similarly, the invention has the advantage that defects only need to be identified at the final stage, but when identified, they can be checked against previous images for defects, either manually by an operator or using damage detection algorithms. This avoids unnecessary processing of large data sets and in the logistics of finished vehicles there is no need to check the vehicle for stains or damage mid-journey unless damage is found during the final inspection stage.

[0037] The optional features of the first to third aspects may be applied to the fourth aspect as well.

[0038] The final stage imaging station may be, for example, a mobile device (e.g., a phone, tablet or similar device) and allowing the operator to manually mark damage on the vehicle image. This can identify defects on stained or soiled vehicles where the use of defect detection algorithms would otherwise result in a large number of false positives.

[0039] In any aspect of the invention, there may be multiple final stage imaging stations located at quality gates in the production and shipping process. For example, at the end of a body shop to check the chassis before it is e-coated, at the end of a paint shop to check for paint and surface defects, to check for defects or deviations in the specifications to which components such as doors must be pre-assembled, at the end of the production line to check for defects or deviations in the specifications required for the finished vehicle, at key handover points in the shipping and logistics journey, and / or at the end of the journey at the dealer handover.

[0040] Defect detection algorithms can be used to identify defects at any end-of-life station. In practice, this may be determined by whether there are repair or rework stations and whether defects need to be identified in real time so that they can be repaired before leaving that stage. Also, if trends are identified using the time it takes a vehicle to go through the entire production and shipping journey, running the defect detection algorithm at additional imaging stations allows for greater accuracy and shorter time intervals between defect detection locations.

[0041] Therefore, the imaging station can preferably be easily configured to run defect detection algorithms as a back-end station, or to operate as a slave and simply record the state of the vehicle using images, physical measurements and other sensor inputs. Advantageously, the hardware configuration is therefore flexible and can be set up as either a slave or a master.

[0042] In any embodiment, the system may be configured to detect defects in a 2D image of a vehicle, output 3D locations of the defects detected in the image, use the 3D locations of the defects to identify the panel, and optionally the area within the panel where the defect is located, and re-project the 3D locations onto multiple 2D views of the vehicle from different viewing angles.

[0043] According to a further aspect of the present invention, a system and method are provided for detecting defects in a 2D image of a vehicle. The system outputs a 3D location of a defect detected in the image, uses the 3D location of the defect to identify the panel, and optionally the area within the panel where the defect is located, and reprojects the 3D location onto multiple 2D representations of views of the vehicle from different viewing angles. This process can be repeated for multiple defects to generate heat maps in the 3D and 2D representations. [Brief explanation of the drawings]

[0044] Specific embodiments of the present invention will now be described, by way of example only, and with reference to the accompanying drawings, in which:

[0045] [Figure 1] FIG. 1 is a diagram illustrating a vehicle production facility. [Figure 2] FIG. 2 is a diagram illustrating a vehicle production facility according to an embodiment of the present invention. [Figure 3] FIG. 3 is a diagram illustrating an example of a vehicle identification device. [Figure 4] FIG. 4 is a diagram of a modular imaging station that can be used as an end-of-sequence imaging station in a vehicle production facility in accordance with one embodiment of the present invention. [Figure 5] FIG. 5 is a diagram of a subset of modules of the modular imaging station of FIG. [Figure 6] FIG. 6 is a flow chart illustrating a computer-implemented quality control method according to one embodiment of the present invention. [Figure 7] FIG. 7 is a diagram illustrating the AI ​​learning and deployment stages of an imaging system according to one embodiment of the present invention. [Figure 8]FIG. 8 is a diagram of a vehicle quality control imaging system distributed across production and distribution sites according to one embodiment of the present invention. [Figure 9] FIG. 9 shows defect heatmaps in 3D and 2D visualization of a vehicle. DETAILED DESCRIPTION OF THE INVENTION

[0046] Figure 1 shows a vehicle production facility PF. The production facility is configured to produce, for example, cars, vans, motorbikes, etc.

[0047] The vehicle production facility PF includes a vehicle production line P1 that includes multiple vehicle production stages P1, P2, P3 arranged to define a production process between an initial production stage P1 and a final production stage P3. While only a few production stages are shown here for ease of explanation, in practice the vehicle production facility PF may have any number of production stages, such as 100 or more, 200 or more, 300 or more, 400 or more, or even 500-600. Furthermore, while the production process is shown as a linear process, the process may have multiple strands of components, such as doors, assembled in a linear production path, with each component's production path flowing in parallel across the vehicle production line PL.

[0048] In this embodiment, the initial production stage P1 is configured to perform a first process that defines a vehicle assembly VA. The final production stage P3 is configured to perform a final process that delivers a completed vehicle VC. One or more intermediate stages P2 are arranged to perform their respective processes on the vehicle assembly VA to develop the vehicle assembly VA into a completed vehicle VC.

[0049] However, the initial production stage P1 may, for example, receive partially assembled vehicles, and the final production stage P3 may output unfinished vehicles, and thus the production process may form part of a larger production process. Furthermore, although the production process is shown as being within a single production facility PF, the production process may be divided among multiple production facilities and may include the distribution of completed vehicles, in which case the final production stage P3 would be a delivery stage.

[0050] Further in FIG. 2, the vehicle production facility PF includes a quality control imaging system IS according to an embodiment of the present invention in addition to a conventional vehicle production line PL.

[0051] The quality control imaging system has a plurality of production imaging stations I1, I2 located between an initial production stage P1 and a final production stage P3. The first production imaging station I1 is positioned to capture one or more production images of a vehicle assembly processed at a production stage. In the illustrated embodiment, the first production imaging station I1 is positioned between the first production stage P1 and the second production stage P2 such that when the vehicle assembly arrives at the first production imaging system I1, the vehicle assembly has been processed at the first production stage P1 but has not yet been processed at the second production stage P2. Similarly, the second production imaging station I2 is positioned between the second production stage P2 and the third production stage P3. Embodiments of the present invention can include any number of production imaging stations, such as one, two, or more.

[0052] Thus, each production imaging station I1, I2 is configured to capture an image of a vehicle assembly VA after processing at a previous production stage and before processing at a next production stage.

[0053] In other embodiments, one or more production imaging stations may be located in parallel with a previous production stage, if appropriate, provided that the vehicle assembly VA is visible for imaging. Additionally, while two production imaging stations are shown, in other embodiments the quality control imaging system may include any number of production imaging stations, and in some embodiments one or more of the production imaging stations may be manual cameras, where a user knows the camera's location when capturing a set of production images and manually transmits the images and identifiers to a second controller through a software application.

[0054] The quality control imaging system also includes a final stage imaging station I3 arranged to acquire multiple evaluation images of the vehicle assembly processed in the final production stage P3. Final stage imaging station I3 may be the vehicle imaging system described with reference to FIG. 3 of WO 2021 / 064351, including a scratch detection camera configured to observe a light imaging background surface and a dark imaging background surface reflected through the vehicle, and a dent detection camera configured to observe a structured light image projected by a structured light source. Thus, final stage imaging station I3 is a relatively complex imaging station compared to the requirements of some or all of the production imaging stations.

[0055] As understood herein, each imaging station can have one or more digital cameras and image acquisition software configured to capture one or more digital images of one or more portions of the vehicle assembly. Preferably, multiple imaging stations can be directed at the same portion of the vehicle assembly as it progresses, so that if a defect is identified by a final-stage imaging station, the defect can be seen in multiple sets of production images. The more imaging stations that can view the same part, the greater the system's resolution for detecting the defect, if it occurs. In one example, the imaging stations are positioned to form an arch surrounding the production line so that all exterior surfaces of the vehicle assembly are imaged. However, in some embodiments, a particular production imaging station can be positioned to face one side of the vehicle assembly, for example.

[0056] The first controller C1 is communicatively connected to the back-end imaging station I3 via a data network, receives the evaluation images acquired by the back-end imaging station I3, and executes a defect detection algorithm using the evaluation images to identify defects in the vehicle, the details of which are described below with reference to FIG.

[0057] The first controller C1 can be connected to the back-end imaging station by a wired data connection, thereby using the wireless network bandwidth of the vehicle production facility PF. Large amounts of data can be transferred without requiring a

[0058] A second controller C2 is communicatively connected to the production imaging stations I1, I2 and configured to store production images of the vehicle assembly.

[0059] The third controller C3 is communicatively connected to the first controller C1 and the second controller C2 and configured to acquire production images from the second controller C1 in response to a defect detection algorithm that identifies defects on the vehicle assembly.

[0060] Thus, the quality control imaging system acquires production images of the vehicle assembly throughout multiple production stages. The production images may, for example, be images of one or more components of the vehicle assembly as it is being assembled. Thus, because the vehicle assembly has been processed by one or more production stages between the acquisition of the first and second production images, the production images define a visual record of the vehicle during assembly, i.e., the second set of production images shows the vehicle assembly in a more completed state than the first set of production images. Once the vehicle assembly is processed through the final production stage, evaluation images are acquired and processed by a defect detection algorithm to identify whether or not a defect is present in the vehicle. Thus, the final stage imaging station is a quality control imaging stage that detects defects such as scratches, dents, dents, and alignment errors. Once a defect is detected by the defect detection algorithm, a production image is acquired. Because the production images show the vehicle assembly at multiple discrete locations along the production stages, the presence or absence of a defect in the production images can be used to identify the production stage in which the defect first occurred, allowing for rapid identification of the source of a quality control issue on the vehicle production line.

[0061] The functions of the three controllers are shown separately here for ease of understanding, but each function may be implemented by a single controller or a distributed computing system, as appropriate. In the illustrated embodiment, the second controller is shown as a single controller connected to each of the production imaging stations, but the second controllers may be distributed. For example, each production imaging station may have a second controller.

[0062] The data network connecting the controller to the imaging station may include, for example, an industrial protocol network such as Modbus or OPC, or a high speed data communication standard connection such as USB, IEEE802 or IEEE1394.

[0063] Each controller may be implemented as a dedicated computer system having a computer processor and non-transitory computer-readable memory that stores computer instructions for performing the functions described herein. At least the first controller C1 may include one or more digital signal processors (DSPs) that analyze large amounts of digital image data.

[0064] The third controller C3 can be communicatively connected to a remote server RS ​​via a wired or wireless data connection, allowing a user to review documentation generated by the third controller C3. The documentation can include, for example, evaluation images and production images showing defects. The remote server can run a dashboard to report defect trends and analysis.

[0065] In some embodiments, at least some of the controllers may be located remotely from the production facility PF. For example, the first and third controllers C1, C3 may be remote from the production facility. Thus, images may be processed locally via a computer in the production facility (e.g., edge computing) or remotely via a computer in the production facility (e.g., edge computing). For example, it can be processed remotely via cloud processing.

[0066] The quality control imaging system IS also includes a vehicle identification system ID configured to identify vehicle assemblies entering each imaging station. As shown in FIG. 3, the imaging system IS can be connected to a production line management system LC and a production line management system PM to obtain information identifying vehicle assemblies approaching each imaging station. For example, a sensor S, such as a light beam or video feed, can detect the presence of a vehicle assembly VA at an imaging station, and the imaging system requests the identification of the vehicle assembly from the production line management system LC, and then requests additional information, such as vehicle type and component identifiers. Alternatively or additionally, the identification system ID can include a camera positioned at each imaging station to read the vehicle identification number VIN on the vehicle assembly. Thus, the quality control imaging system can track the vehicle assembly as it passes through the system. By knowing the location and orientation of the imaging stations relative to the vehicle assembly and the production line, the system can know which components of the vehicle assembly are visible to each camera or other information gathering device of the imaging system IS, and this information can be associated with the images and measurements captured at each imaging station.

[0067] Continuing with reference to FIG. 4 , back-end imaging station 14 may employ a modular configuration including a first blemish detection module D, a second blemish detection module L, and a dent detection module S. Each blemish detection module may include a frame F, an imaging background surface mounted on the frame F, and one or more first blemish detection cameras oriented to view the imaging background surface in reflection by the vehicle assembly as the dent detection modules move through the module mounted on the frame. The imaging background surface of the first blemish detection module may be relatively light or dark compared to the imaging background surface of the second blemish detection module. In one example, the background surface of the first blemish detection module may be illuminated with white light, while the imaging background surface of the second blemish detection module may be gray or black and may be a non-reflective and / or non-illuminated surface. The dent detection module S may include a frame F, a structured light source mounted on the frame and positioned to generate a structured light image, and one or more dent detection cameras mounted on the frame and oriented to view the structured light image on the vehicle assembly as the dent detection modules move through the module. An alignment module (not shown) may also be provided for "gap and flash" testing.

[0068] The frames F of the first scratch detection module, the second scratch detection module, and the dent detection module can include standardized mounting points so that each module can be quickly and easily connected to any other module at any stage. In one example, the frames can include mounting holes that align coaxially when two modules are placed end-to-end. The frames can be, for example, laser-cut metal to improve connection accuracy between the modules.

[0069] Each module may include a module controller that, in addition to controlling the module's components such as the camera and light source, functions as or forms part of a first controller C1 or functions as one of the second controllers C2, making each module D, L, S essentially "plug and play." Modular systems may be useful in terms of ease of installation, planning, and interoperability.

[0070] Each module has a light control element that extends around the sides and top of the module and passes through the module. The modules may include an arch that defines a path for the vehicle assembly. When two or more modules are attached together, their respective covers interlock to form a continuous, light-controlled vehicle assembly path through the imaging station. In such an embodiment, it is possible to control the light to image the sides and top of the vehicle assembly.

[0071] In any embodiment, the camera may comprise one or more scan cameras, such as a Hikvision (RTM) MV-CA050-10GC area scan camera.

[0072] 5, some or all of production imaging stations I1-I3 can each be comprised of one or two modules present in back-end imaging station I4. Thus, production imaging stations I1-I3 can be simpler in terms of mechanical footprint / size and / or than back-end imaging stations, thereby reducing the overall size and cost of the quality control imaging system while providing high-resolution detection of production stages causing defects. In this example, production imaging station I1 is comprised of modules L and S.

[0073] Each module can be arranged to be movably positioned, so that the imaging station modules can be easily moved from one point to another in the production line to change the production imaging configuration and move the production imaging station to locations where defects are expected.

[0074] In some embodiments, the quality control imaging station IS can have a controller configured to receive the production images and execute a defect location algorithm to identify the production imaging station where the defect was first identified. For example, this can be performed by a first controller C1. The defect location algorithm can be trained and developed in the same manner as the defect detection algorithm.

[0075] 6, a computer-implemented method for assessing the quality of vehicles produced on a vehicle production line is indicated generally at 100. The production line includes a plurality of individual vehicle production stages arranged to define a production run between an initial production stage and a final production stage.

[0076] In step 102, the method includes identifying a vehicle assembly at a first production imaging station, obtaining a first set of one or more production images of the vehicle assembly following processing in a first production stage, and storing the first set of production images.

[0077] In step 104, the method includes identifying the vehicle assembly at a second production imaging station, obtaining a second set of one or more production images of the vehicle assembly after processing at a second production stage different from the first production stage, and storing the second set of production images.

[0078] In step 106, the method includes identifying the vehicle assembly at a back-end imaging station and obtaining one or more evaluation images of the vehicle assembly after processing in the back-end production stage.

[0079] In step 108, the method includes executing a defect detection algorithm that uses the evaluation images from the back-end imaging station to identify defects on the vehicle.

[0080] In step 110, the method includes: identifying a defect on the vehicle by a defect detection algorithm; acquiring production images from the first and second production imaging stations in response to the defect detection algorithm identifying a defect.

[0081] In step 112, the method includes the optional step of generating a report that includes at least one evaluation image showing the defects and a production image.

[0082] In step 114, the method includes the optional step of deleting the production image of the vehicle assembly if the defect detection algorithm does not identify any defects on the vehicle assembly.

[0083] The method may also include, prior to step 102, the optional step of rearranging one or more of the production imaging stations.

[0084] Referring now to FIG. 7, there is shown a system diagram illustrating the AI ​​training process 130 and deployment process 140 of the system according to the present embodiment.

[0085] The training process 130 comprises a data and pre-processing module 132, an AI algorithm module 134, a learning algorithm module 136, and the architecture / platform underlying the execution of the training process 130.

[0086] In the data and preprocessing module 132, training images of defective vehicles are provided to illustrate what the system seeks to identify and quantify. For example, images with visible and labeled scratches and / or dents are provided. For each type of damage, a severity can be labeled to allow the AI ​​to estimate both the type of damage and the extent of the damage. Preferably, the images have a resolution where 1 millimeter on the vehicle corresponds to approximately 3 or 4 pixels in the corresponding image of the vehicle. In one exemplary implementation, this preferred resolution can be achieved with 64 megapixel images. The training data also includes label information corresponding to areas of the vehicle where instances of damage are located. The labeled areas associated with the images can correspond to bounding boxes that define areas of the image that contain damage. Thus, the label information can be used to identify, for a given image, the boundaries of the bounding box (e.g., the top left corner of the region) and the area of ​​the image that contains the damage. The relative XY positions of the corners are expressed along with the width and / or height of the region) and the dimensions of the area contained within the region. and labels corresponding to classes of damage (e.g., scratches, dents, chips, etc.) that are included in the training data. Each image in the training data can be associated with multiple labeled regions. In one exemplary implementation, the training data includes 500 manually annotated images of defective vehicles, each image including one or more labeled regions associated with either a scratch class, a dent class, or a chip class.

[0087] The AI ​​algorithm module 134 may include known algorithms such as convolutional neural networks (CNNs), support vector machines (SVMs), and the like.

[0088] The learning algorithm module 136 applies the training data 132 to an AI algorithm 143. If the AI ​​algorithm is CNN-based, for example, the learning algorithm may include backpropagation with stochastic gradient descent. If the AI ​​algorithm is SVM-based, the learning algorithm may include the use of known methods such as quadratic programming.

[0089] The AI ​​learning platform may comprise any suitable conventional computing device, including, for example, one or more GPUs, and may be implemented as a distributed network of commuting devices.

[0090] The development stage 140 forms an integral part of the vehicle imaging station 1 and comprises a novel data module 142 , a model module 144 and a prediction module 146 .

[0091] The model module 144 includes the learned algorithm output from the learning algorithm module 136 and executed on the data processor 42, but may alternatively be executed by a data processor on a server, for example.

[0092] The model module 144 receives as input the defect assessment images from the new data module 142. The trained model is thus a program that can be executed to identify defects in a vehicle using the acquired images.

[0093] The model module 144 outputs a prediction 146 that includes one or more of the instance and type of damage, the extent of damage, and the location of damage.

[0094] In one example, the defect multi-tasking CNN uses a machine learning dataset to assess the vehicle condition and provide a probability of damage, a class of damage, and a size of damage. The multi-tasking CNN can operate locally on a data processor associated with the imaging station and / or in cloud-based computing, such as on a server. The multi-tasking CNN can continue to grow and learn using images acquired within the system. In one example, a trained neural network can be expanded using new data. Alternatively, the model can be updated by re-training the entire model, possibly using a trained model as a starting point; i.e., rather than starting with a completely random configuration of network weights, pre-trained weights are used as a starting point.

[0095] In one particular exemplary implementation, the AI ​​algorithm module 134 implements the YOLO algorithm trained on a dataset such as that described above in connection with the data and preprocessing module 132. The YOLO algorithm includes a CSPDarknet53 backbone and a YOLOv5 object detector. The architecture is a convolutional base layer with cross-stage partial blocks that split and merge feature maps in the base layer. The hyperparameters of the YOLO algorithm were determined using a standard grid search approach, and the algorithm is optimized using an Adam optimizer with ε=1e -7 and were trained with β1=0.9 and β1=0.999.

[0096] Images can be uploaded to cloud storage, where they can be retrieved by clients from anywhere in the world. Once all images are uploaded, they are delivered to the AI ​​via a worker queue, where a service running the AI ​​processes the images as they are uploaded. The AI ​​is applied directly to the vehicle images. The machine learning model can consist of a convolutional neural network with the YOLO object detection architecture, trained on hundreds of thousands of previous examples of vehicle defects so that the ImageNet backbone can "learn" what constitutes a defect and what does not. The AI ​​model can be deployed on powerful cloud servers using state-of-the-art GPU computing, which processes the images and returns detection results as coordinate-based bounding boxes for display on a front-end dashboard.

[0097] 8 illustrates a vehicle quality control imaging system IS' distributed across production and distribution sites according to an embodiment of the present invention. The imaging system IS' is similar to the imaging system IS described above, and for simplicity, the following description focuses on the differences between the systems.

[0098] In this embodiment, the production facility PF includes a first strand of production stations P1'-P3' and a second strand of production stations P1-P3 that feed stations P4 and P5 of a common production line. The first strand can assemble a first component, such as a door, and the second strand can assemble a different component, such as a hood. Imaging stations I1'-I3' and I1-I5 are provided between the above manufacturing stations. In this example, imaging stations I1' and I2' are used to image the first component. I3' is a production imaging station for recording images of components being assembled on the strand, and I3' is a final stage imaging station for quality control of the first component. A repair station (not shown) can be located between the imaging station I3' and the next production station P4. Stations I1 and I2 may be production imaging stations for recording images of components being assembled on the second strand, and I3 may be a final stage imaging station for quality control of the second components. The station (not shown) can be placed between the imaging station I3 and the next production station P5. Multiple further production strands can be equipped with imaging stations arranged as desired, and a common or "main" production line can be equipped with a production imaging station such as I4 and a back-end imaging station such as I5. Quality control imaging can therefore be performed at various stages in the manufacturing process.

[0099] The system IS′ in this embodiment also includes an imaging station 16 for the first shipping stage D1 at a first shipping location L1. The first shipping stage may be, for example, an off-site location where vehicles are stored before being loaded onto a ship for overseas transport. Similarly, the system includes an imaging station 17 for the second shipping stage D2 at a second shipping location L2. The second shipping stage may be, for example, a location where vehicles are stored after being unloaded from a ship. Similarly, the system includes an imaging station 18 for the third shipping stage D3 at a third shipping location L3. The third shipping stage may be, for example, a dealership. The imaging stations 16 and 17 may be production imaging stations, and the imaging station 18 may be a final-stage imaging station.

[0100] The back-end imaging station 18 may have a mobile device (e.g., phone, tablet, etc.) and an application that allows an operator to manually mark damaged areas on the vehicle image, thereby identifying defects on stained or soiled vehicles where defect detection algorithms may produce a large number of false positives.

[0101] Therefore, the technical concept of the present invention can be widely applied to the overall stages of vehicle production and shipping, including the production stage, the shipping stage, or both.

[0102] 9, in any embodiment, the back-end imaging station can be configured to identify which vehicle components are defective and the location of the defects on the components from the evaluation images. A dedicated controller may be provided for this purpose. In one example, a localization module of the system converts pixel coordinates of defects detected in the evaluation images into three-dimensional positions (x, y, z) within the vehicle's frame of reference; It assigns attributes such as predetermined panel names and zone names. The calibration module can be composed of an intrinsic calibration module, an extrinsic calibration module, and a depth map generation module. The calibration process is performed once per system and vehicle model and is used to identify camera parameters and generate an offline 3D representation of the vehicle at the camera's trigger position as the vehicle enters the system. The intrinsic parameter calibration module calculates the camera's intrinsic parameters such as focal length, optical center, and distortion coefficients. The extrinsic calibration module The module calculates the camera's position and orientation relative to the system's reference frame, for example, by using keypoints selected as ground truth on an exact replica of the vehicle in a 3D simulation environment and the same keypoints selected for verification on images acquired from the system. The depth map generation module uses the intrinsic and extrinsic parameters to generate a depth map with the 3D location of each point on the vehicle's 3D model for each camera in the system and for each possible position of the vehicle within the scanned area. Next, in the unfolding phase, the ray casting module loads all the information generated during the calibration phase and outputs the 3D location of defects detected on the image. The module positions the 3D model of the vehicle at the estimated location based on the camera trigger, casts rays from the camera through the pixel coordinates of the image into 3D space, and returns the defect location information from the depth map. The system can create a 3D visual representation M of the vehicle showing the defects. The 3D visual representation M can be rotated by a user, for example, a user viewing the visual representation on a remote server RS. The panel identification module then uses the 3D location of the defect to identify the panel and zone within the panel where the defect is located. The module uses different viewing angles ( Reproject the 3D position onto a 2D image D of the vehicle's view from, for example, left, right, front, rear, top. This process can be repeated for multiple vehicles to create a heat map of defects and their locations over a period of time. The heat map can include a three-dimensional representation M and zones Z in the two-dimensional image D that visually indicate the number and location of defects by color, size, etc. A defect location algorithm can be used to indicate where the system estimates the defects are located. Accordingly, embodiments of the present invention extend to systems and methods that detect defects in a two-dimensional image of a vehicle, output three-dimensional locations of the defects detected in the image, use the three-dimensional locations of the defects to identify the panel and optionally the zone within the panel that is defective, and reproject the three-dimensional locations onto multiple two-dimensional images of views of the vehicle from different viewing angles. Such embodiments do not require any production imaging stations or second and third controllers; i.e., the systems and methods can simply use images of the vehicle at a single location, such as a back-end imaging station.

[0103] Although the present invention has been described above with reference to one or more preferred embodiments, it will be understood that various changes or modifications can be made thereto without departing from the scope of the invention as defined in the appended claims. In this specification, the word "comprises" means "includes" or "consists of," and therefore does not exclude the presence of elements or steps other than those listed in any claim or the specification as a whole. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. 1. A computer-implemented method for evaluating quality of vehicles produced on a vehicle production line, comprising: the production line having a plurality of individual vehicle production stages arranged to define a production run between an initial production stage and a final production stage; The method comprises: identifying a vehicle assembly at a first production imaging station, obtaining a first set of one or more production images of the vehicle assembly after processing at a first production stage, and storing the first set of production images; identifying the vehicle assembly at a back-end imaging station and acquiring one or more evaluation images of the vehicle assembly after the back-end production processing; running a defect detection algorithm using the evaluation image to identify defects on the vehicle; acquiring the production image in response to identifying a defect on the vehicle by the defect detection algorithm; 10. A computer-implemented method comprising:

2. 10. The method of claim 1, further comprising identifying the vehicle assembly, obtaining one or more additional sets of one or more production images of the vehicle assembly after additional separate production stage processing, and storing the additional sets of production images.

3. 3. The method of claim 1 or 2, wherein one or more of the sets of production images each have a smaller number of images than the set of evaluation images and / or have images of lower resolution than the set of evaluation images.

4. 4. The method of claim 1, further comprising, after the step of acquiring the production image in response to the defect on the vehicle being identified by the defect detection algorithm, generating a report including the production image and at least one evaluation image showing the defect.

5. 5. The method of claim 1, further comprising, after the step of acquiring the production images in response to the defect being identified on the vehicle by the defect detection algorithm, searching the production images for the defect identified in the evaluation images to identify the first set of production images that show the defect, and optionally generating a report that identifies the production stage before the defect first occurs in a set of production images.

6. 6. The method of claim 1, further comprising the step of associating each set of production images with the production stage at which the vehicle assembly was last processed when the images were acquired.

7. 7. The method of claim 1, further comprising deleting the production image if the defect detection algorithm does not identify any defects on the vehicle.

8. 8. The method of claim 1, wherein the step of identifying the vehicle includes obtaining and storing a vehicle assembly identifier with each of the set of production images and the set of evaluation images.

9. 9. The method of claim 1, further comprising an initial step of moving a production imaging station from a first position during a first pair of production stages to a second position during a second pair of production stages separate from the first pair.

10. A vehicle production facility, a vehicle production line having a plurality of individual vehicle production stages arranged to define a production run between an initial production stage and a final production stage; 1. A quality control imaging system, comprising: one or more production imaging stations located along the production process between the initial production stage and the final production stage, each production imaging station configured to capture a set of one or more production images of a vehicle assembly processed at a production stage; a back-end imaging station positioned to acquire a set of multiple evaluation images of the vehicle assembly processed at the back-end production stage; an identification system arranged to identify a vehicle assembly entering each imaging station; a first controller communicatively connected to the back-end imaging station, the first controller configured to receive the evaluation images acquired by the back-end imaging station and to execute a defect detection algorithm using the evaluation images to identify defects in the vehicle; a second controller communicatively connected to the production imaging station, the second controller configured to store the production images; a third controller communicatively coupled to the first controller and the second controller, the third controller configured to acquire the production image in response to the defect detection algorithm identifying a defect on the vehicle; a quality control imaging system having A vehicle production facility comprising:

11. 11. The vehicle production facility of claim 10, wherein the back-end imaging station has a modular configuration and includes a first surface defect detection module, a second surface defect detection module, and a dent detection module, each surface defect detection module including a frame, an imaging background surface mounted on the frame, and one or more first surface detection cameras mounted on the frame and oriented to view the imaging background surface in reflection by the vehicle assembly as the vehicle assembly moves through the module, the imaging background surface of the first surface detection module being relatively brighter or relatively darker than the imaging background surface of the second surface detection module, and the dent detection module includes a frame, a structured light source mounted on the frame and positioned to generate a structured light image, and one or more dent detection cameras mounted on the frame and oriented to view the structured light image on the vehicle assembly as the vehicle assembly moves through the module.

12. 12. The vehicle production facility of claim 11, wherein the frame of the first surface detection module, the frame of the second surface detection module, and the frame of the dent detection module include standardized mounting points so that each module can be quickly and easily connected to any other module at any stage, and / or each module includes a module controller configured to function as one of the first controller or the second controller.

13. Each module has a perimeter that extends around the sides and top of the module.

13. A vehicle production facility according to claim 11 or 12, comprising an archway defining the path of light-controlled vehicle assemblies therethrough.

14. 14. The vehicle production facility of claim 11, wherein each of the one or more production imaging stations is comprised of one or more modules present in the back-end imaging station, and optionally each module is movably positioned relative to the production line.

15. 15. The vehicle production facility of claim 1, wherein the first controller is connected to the back-end imaging station by a wireless connection.

16. 16. A quality control imaging system for use in a vehicle production facility according to any one of claims 10 to 15, comprising: one or more production imaging stations located along the production process between the initial production stage and the final production stage, each production imaging station configured to capture a set of one or more production images of a vehicle assembly processed at a production stage; a back-end imaging station positioned to acquire a set of evaluation images of the vehicle assembly processed at said back-end production stage; an identification system arranged to identify a vehicle assembly entering each imaging station; a first controller communicatively connected to the back-end imaging station, the first controller configured to receive the evaluation images acquired by the back-end imaging station and to execute a defect detection algorithm using the evaluation images to identify defects in the vehicle; a second controller communicatively connected to the production imaging station, the second controller configured to store the production images; a third controller communicatively coupled to the first controller and the second controller, the third controller configured to acquire the production image in response to the defect detection algorithm identifying a defect on the vehicle; A quality control imaging system comprising:

17. 1. A computer-implemented method for assessing the quality of a vehicle as it passes through a plurality of distinct journey stages between an initial stage and a final stage, comprising: identifying a vehicle assembly at a first journey imaging station, acquiring a first set of one or more journey images of said vehicle after a first operational stage of processing, and storing said first set of journey images; identifying the vehicle at a final stage imaging station and acquiring one or more evaluation images of the vehicle after processing at the final journey stage; running a defect detection algorithm using the evaluation image to identify defects on the vehicle; acquiring the journey image in response to the defect detection algorithm identifying a defect on the vehicle; 10. A computer-implemented method comprising:

Images