Machine vision and robot installation system and method

Abstract

The present invention relates to machine vision and robotic installation systems and methods. The machine vision method and system determine whether an object in a work area has one or more predetermined features. The method includes capturing image data of the work area, applying a filter to the image data, wherein the filter includes aspects corresponding to specific features, and determining whether the object has the specific features based at least in part on the application. The system includes: a camera system configured to capture image data of the work area; and a controller, communicatively coupled to the camera system, and including a non-transitory computer-readable medium having computer-readable instructions, which when executed causes the controller to apply the filter to the image data, and determine whether the object has the specific features based at least in part on the application of the filter. A robotic installation method and system utilizing the machine vision method and system are also disclosed.

Description

Technical Field

[0001] The present disclosure relates to machine vision. Background Art

[0002] Manufacturing is becoming increasingly automated. Current robotic manufacturing systems are very effective at repetitive tasks; however, they do not necessarily adapt to changing environments or errors introduced by humans or other upstream inputs. For example, a nut-running robot (e.g., a robotic arm with a nut-installing end effector) responsible for installing nuts onto threaded fasteners may install nuts onto threaded fasteners that are the incorrect size or configuration, causing humans to not only identify the error but also subsequently correct the error. Summary of the invention

[0003] A machine vision and robotics installation system and method are disclosed.

[0004] An automated machine vision method for determining whether an object in a work area has one or more predetermined features according to the present disclosure includes: capturing image data of the work area from a camera system, applying a filter to the image data to produce filtered data, wherein the filter includes aspects of specific features corresponding to the one or more predetermined features, and determining whether the object has the specific features based at least in part on the application. A robotic mounting method according to the present disclosure includes performing an automated machine vision method according to the present disclosure, wherein the camera system is mounted to, with, or as an end effector of a robotic arm, and based on the determination, using the robotic arm to mount a component in a predetermined configuration relative to the object.

[0005] A machine vision system for determining whether an object in a work area has one or more predetermined features according to the present disclosure includes: a camera system configured to capture image data in the work area, and a controller communicatively coupled to the camera system to receive image data from the camera system. The controller includes a non-transitory computer-readable medium having computer-readable instructions, which, when executed, causes the controller to apply a filter to the image data to produce filtered data, wherein the filter includes aspects of specific features corresponding to one or more predetermined features, and determines whether the object has specific features based at least in part on the application of the filter. A robot mounting system according to the present disclosure includes a machine vision system according to the present disclosure and a robotic arm, wherein the camera system is mounted to, mounted with, or mounted as an end effector of the robotic arm. The robotic arm is configured to mount a component in a predetermined configuration relative to the object based in part on a determination made by the controller that the object has specific features. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a flow chart schematically illustrating a method according to the present disclosure.

[0007] Figure 2 is a diagram schematically showing a system according to the present disclosure.

[0008] Figure 3 is a rough schematic diagram representing an example robotic mounting system according to the present disclosure.

[0009] Figure 4 is a flow chart schematically representing an aircraft production and service method.

[0010] Figure 5 is a block diagram schematically representing an aircraft. DETAILED DESCRIPTION

[0011] Typically, in the accompanying drawings, elements that may be included in a given example are shown in solid lines, while optional elements for a given example are shown in dashed lines. However, elements shown in solid lines are not necessary for all examples of the present disclosure, and elements shown in solid lines may be omitted from a particular example without departing from the scope of the present disclosure.

[0012] Machine vision and robotics installation systems and methods are disclosed herein. Figure 1 A flow chart schematically illustrating an automated machine vision method 100 for determining whether an object 302 in a work area 304 has one or more predetermined features 306 is provided, and Figure 2 A schematic representation of a machine vision system 300 for determining whether an object 302 in a work area 304 has one or more predetermined features 306 is provided. As will be appreciated from the discussion herein, Figure 1 and Figure 2 The schematic representations of the method 100 and system 300 in the figures are not limiting, and other methods 100, steps of the method 100, systems 300, and elements of the system 300 are within the scope of the present disclosure, including methods 100 having more or less steps than the number shown, and systems 300 having more or less elements than the number shown. It will also be understood from the discussion herein that the method 100 need not be performed in the order shown. Figure 1 The system 300 may be described as being configured to perform or implement the method 100 according to the present disclosure; however, not all methods 100 need to be performed or implemented by the system 300 according to the present disclosure, and not all systems 300 need to perform or implement the method 100 according to the present disclosure.

[0013] As described above, the method 100 and system 300 determine whether the object 302 in the work area 304 has one or more predetermined features 306 (e.g., physical or structural features or a specific location or orientation of the object 302). Therefore, by confirming that the features 306 of the object 302 are features that are expected to be present, or otherwise correspond to an object that is expected to be present at the location of the object 302, the method 100 and system 300 can determine whether the object 302 is the correct object, for example, whether it is ready to be processed by subsequent operations. Additionally or alternatively, the method 100 and system 300 can determine whether the object 302 is in a desired configuration, such as fully installed, correctly positioned, assembled with different components, etc. Conversely, if the method 100 or system 300 determines that the features of the object 302 are not features that are expected to be present, or otherwise correspond to an object or configuration thereof that is expected to be present at the location of the object 302, the subsequent operation can be avoided, or an alternative operation can be performed, such as by replacing the object 302 with the correct object, by further manipulating the object 302 before subsequent processing, etc. Thus, some methods 100 and systems 300 may be described as quality control methods and systems.

[0014] Method 100 and system 300 may be described as automatic methods and systems if, upon initiation or activation, the steps required to achieve the ultimate goal of method 100 or system 300 may be fully automated by the system or its components without requiring external input from a human user.

[0015] Work area 304 can be any suitable work area, such as in a manufacturing or production environment. As an illustrative, non-exclusive example, work area 304 can include a product or a portion thereof, such as a vehicle, aircraft, or machine, during an assembly or manufacturing stage. As a more specific example, work area 304 can include an area of ​​a product where fasteners are installed to assemble two or more parts together. In such an example, object 302 can include fasteners; however, method 100 and system 300 can be implemented or configured with any suitable configuration of object 302.

[0016] Specific reference Figure 1 And overall reference Figure 2, method 100 includes capturing (102) image data of a work area 304 from a camera system 308, applying (104) a filter to the image data to produce filtered data, wherein the filter includes an aspect corresponding to a particular feature of one or more predetermined features 306, and determining (106) whether the object 302 has the particular feature based at least in part on the applying (104). In other words, image data associated with the work area 304 is captured, a filter having an aspect corresponding to the particular feature is applied to the image data, and filtered image data is formed, and it is determined whether the object 302 has the particular feature. As an illustrative, non-exclusive example, when the object 302 is a threaded fastener, the filter may have an aspect corresponding to a thread configuration (e.g., a pitch) of the object in which the object 302 is expected to be positioned, and as a result of the filtering, it is determined whether the threaded fastener has the expected pitch.

[0017] Specific reference Figure 2 , the system 300 includes a camera system 308 configured to capture image data of a work area 304, and a controller 310 communicatively coupled to the camera system 308 to receive the image data from the camera system 308. The controller 310 may be described as being configured to perform or implement the method 100 according to the present disclosure. Additionally or alternatively, the controller 310 may include a non-transitory computer-readable medium having computer-readable instructions that, when executed, cause the controller to apply a filter to the image data to create filtered data, wherein the filter includes specific features corresponding to the one or more predetermined features 306, and based at least in part on the application of the filter, determine whether the object 302 has the specific features.

[0018] The controller 310 may be any suitable device or devices configured to perform the functions of the controller 310 discussed herein. For example, the controller 310 may include one or more of an electronic controller, a dedicated controller, a special purpose controller, a personal computer, a special purpose computer, a display device, a logic device, a storage device, and / or a storage device having a computer-readable medium suitable for storing computer-executable instructions for implementing various aspects of the system and / or method according to the present disclosure. Additionally or alternatively, the controller 310 may include or be configured to read a non-transitory computer-readable memory or storage for implementing the steps of the method or method according to the present disclosure, a medium suitable for storing computer-executable instructions or software. Examples of such media include CD-ROMs, disks, hard drives, flash memory, etc. As used herein, storage or memory, devices and media having computer-executable instructions, and computer-implemented methods and other methods according to the present disclosure are considered to be within the scope of subject matter that is considered patentable.

[0019] In some examples of the method 100 and system 300, the camera system 308 is positioned at a known or determined position and orientation relative to the working area 304. For example, from the perspective of the camera system 308, the size, shape, and orientation of the object 302 affect the captured image data (e.g., the set of pixels corresponding to the object 302), and therefore affect the corresponding analysis of the image data to determine whether the object 302 has a particular feature. Therefore, the coordinate systems associated with the camera system 308 and the working area 304 need to be aligned or at least coordinated in a known manner in order to implement the method 100 and system 300 according to these example operations. Therefore, some methods 100 may be described as further including determining or obtaining the position and / or orientation of the camera system 308 relative to the working area 304, and / or vice versa, and similarly, in some systems 300, the computer-readable instructions, when executed, may further cause the controller to determine or obtain the position and / or orientation of the camera system 308 relative to the working area 304, and / or vice versa.

[0020] In some such examples, one or both of the camera system 308 and the working area 304 may be located at a known or determined location, e.g., the controller 310 of the system 300 considers the known or determined location when determining whether the object 302 has one or more predetermined features. This may include determining and / or establishing a coordinate system that describes the location and / or orientation of the system 300, determining and / or establishing a coordinate system that describes the location and / or orientation of the working area 304, determining and / or establishing a coordinate system that describes the location and / or orientation of the camera system 308, and / or determining and / or establishing a relationship, offset, and / or difference between two or more coordinate systems.

[0021] For example, one or both of the camera system 308 and the work area 304 can be associated with or monitored by a positioning system, such as an indoor positioning system in a manufacturing or production environment, which is configured to detect the precise location of the camera system 308 and / or the work area 304. Additionally or alternatively, the camera system 308 can be fixed in space and the work area 304 can be moved into the field of view of the camera system 308, for example, the work area 304 can be precisely located by a precision tool, or loosely located by a global resynchronization operation performed by the system 300 using a vision system or a touch probe. Alternatively, the work area 304 can be fixed in space and the camera system 308 can be moved so that the work area 304 is in the field of view of the camera system 308, for example, the camera system 308 can be precisely located by a precision tool, or loosely located by a global resynchronization operation performed by the system 300 using a vision system or a touch probe. Alternatively, a mathematical model or description of the work area 304 can be used to estimate, establish and / or determine the relative position between the camera system 308 and the work area 304. For example, the mathematical model or description of the work area 304 may include and / or be a CAD model of the work area 304 or a two-dimensional projection of the CAD model of the work area 304 .

[0022] In some examples of the method 100 and system 300, the camera system 308 is supported by a camera support 312 (e.g., a robotic arm). That is, the camera system 308 can be mounted to, with, or as an end effector of the robotic arm. In some such systems 300, the controller 310 is communicatively coupled to the camera support to receive the position and orientation of the camera system 308 relative to the work area 304. As discussed, considering, knowing, determining, and / or quantifying the position and orientation of the camera system 308 relative to the work area 304 facilitates analyzing the image data to determine whether the object 302 has a particular feature.

[0023] In some examples of method 100 and system 300, the image data includes color data. For example, feature 306 can be a specific color. However, in applications where the color of an object is not required or otherwise indicates whether object 302 corresponds to an expected object, the image data may not include color data. In such examples, camera system 308 can therefore capture black and white or grayscale data.

[0024] Some methods 100 also include illuminating (108) the working area 304 during the capturing (102), such as Figure 1 Thus, some systems 300 may further include an illumination device 314 configured to illuminate the working area 304 when the camera system 308 is capturing image data, such as Figure 2For example, illumination device 314 may include a flash device that is communicatively coupled to camera system 308 or a component thereof such that the flash illuminates work area 304 when camera system 308 captures image data.

[0025] like Figure 1 As schematically represented in , some methods 100 also include, after capturing (102) the image data and before applying (104) the filter, subtracting (110) data corresponding to the portion of the work area 304 that is not the object 302 from the image data based on the known color of the portion of the work area 304 that is not the object 302. In the corresponding system 300, the computer-readable instructions, when executed, thus further cause the controller to subtract the data corresponding to the portion of the work area 304 that is not the object 302. For example, in some applications, the color of the work area 304 may be uniform except for the one or more objects 302 present in the work area 304. As a more specific example, the work area 304 may include a sheet of material in an original or painted state, wherein the original or painted state has a specific color. Therefore, the data associated with the specific color can be subtracted from the image data, thereby retaining only or generally retaining only the data associated with the one or more objects 302 in the work area 304. Subtracting the color data additionally or alternatively can be described as chroma keying.

[0026] Some methods 100 also include converting (112) the image data to an HSV (hue, saturation, value) domain after capturing (102) the image data and before applying (104) the filter. In a corresponding system 300, the computer-readable instructions, when executed, further cause the controller to convert the image data to the HSV domain before applying the filter. Although not required in all methods 100 and systems 300, converting the image data to the HSV domain can help to subtract color data associated with portions of the working area 304 that are not the object 302.

[0027] Additionally or alternatively, some methods 100 also include, after capturing (102) the image data and before applying (104) the filter, binary thresholding (114) the image data based on the assumed characteristics of the one or more predetermined characteristics 306, and identifying (116) one or more groups of pixels as candidates for the assumed characteristics in response to the binary thresholding (114). In the corresponding system 300, the computer-readable instructions, when executed, thus further cause the controller to binary threshold the image data based on the assumed characteristics of the one or more predetermined characteristics 306 before applying the filter, and identifying one or more groups of pixels as candidates for the assumed characteristics in response to the image data being binary thresholded. As illustrative, non-exclusive examples, the assumed characteristics may be one or more of a color, a shape, a size, a marking, and a thread configuration associated with an object expected or desired in the work area 304. In some examples, the assumed characteristics correspond to the expected or desired objects in the work area 304 based on a database associated with the work area 304. For example, a product being assembled or manufactured may have a parts list stored in a database in which the characteristics of the parts are identified. Thus, if an object within the working area 304 is expected or desired to have an assumed characteristic, such as a particular color, then binary thresholding (114) based on the assumed characteristic helps identify a subset of pixels or image data as candidates for the assumed characteristic. As a more specific example, a fastener or portion of a fastener intended for a particular component of a product may have a particular color. By binary thresholding the image data based on the particular color, a particular group of pixels corresponding to the particular color is identified in the image data for further processing to ultimately determine whether the object 302 is actually the object that is expected to be present, or whether the object 302 is in a desired configuration, such as fully installed, properly positioned, assembled with different components, etc.

[0028] Some such methods 100 also include noise filtering (118) the image data after the binary thresholding (114) (i.e., applying a noise filter (e.g., a binary noise filter)), in which case identifying (116) one or more groups of pixels is also responsive to the noise filtering (118). In the corresponding system 300, the computer-readable instructions, when executed, therefore further cause the controller to noise filter the image data after the image data is binary thresholded (i.e., applying a noise filter (e.g., a binary noise filter)), and identifying one or more groups of pixels is also responsive to the image data being noise filtered. For example, depending on whether the working area 304 is illuminated and / or depending on the degree or quality of illumination of the working area 304, the image data after the binary thresholding (114) may be noisy, and the noise filtering (118) may clean up the image data and help ultimately determine whether the object 302 is actually the object that is expected to be present, or whether the object 302 is in a desired configuration, such as fully installed, properly positioned, assembled with different components, etc.

[0029] In some methods 100 and systems 300, the particular feature whose presence is determined and corresponds to one aspect of the filter applied to the image data when generating the filtered image data is an assumed texture of one or more predetermined features. In some such examples, the assumed texture corresponds to an expected or desired object within the work area 304 based on a database associated with the work area 304. For example, and as discussed, a product being assembled or manufactured may have a parts list stored within a database in which the features of the parts are identified. Therefore, if an object in the work area 304 is expected or expected to have an assumed feature, such as a particular texture, the application of the filter helps to ultimately determine whether the object 302 has the assumed texture. As a more specific example, a fastener intended for a particular component of a product may have threads that cause the fastener to have a texture corresponding to a particular thread size or pitch.

[0030] In some examples of the method 100 and system 300, the filter is a Gabor filter. For example, for a thread pitch associated with a desired fastener, and based on relative and / or absolute position and / or orientation data associated with the working area 304 and / or the camera system 308 (e.g., known or acquired during the method 100 and / or the system 300), an appropriate Gabor filter can be selected or derived and applied to the image data, such as one or more groups of pixels identified as a result of the binary thresholding (114). The greater the response of the Gabor filter to the group of pixels, the greater the confidence that the group of pixels actually represents a feature (e.g., a particular thread pitch) of an object desired in the working area 304.

[0031] As an illustrative, non-exclusive example, the desired fastener may have a known pitch, which may be defined along the longitudinal axis of the known fastener. The position of the object 302 in the system 300, in the working area 304 and / or relative to the camera system 308 may be known and / or obtainable (e.g., as discussed herein, during the method 100 and / or using the system 300). The known pitch exhibiting a corresponding consistent wavelength may be scaled based on the position of the object 302, such as by a perspective projection of the object 302, to generate a scaled wavelength. The scaled wavelength may then be used to create, generate and / or select a Gabor filter or a directional Gabor filter family. The Gabor filter or directional Gabor filter family may then be applied to the image data. If the image data includes a desired fastener with a known pitch (i.e., if the object 302 is the desired fastener), the Gabor filter will exhibit a strong response. On the contrary, if the image data does not include a desired fastener with a known pitch (i.e., if the object 302 is not the desired fastener), the Gabor filter will exhibit a weak or relatively weak response.

[0032] As another illustrative, non-exclusive example, the longitudinal axis of the object 302 may be oriented at an angle about or relative to the imaging plane of the camera system 308. Under these conditions, the scaled wavelength described above may also be multiplied by an angle correction factor to produce and / or generate an angle-corrected and scaled wavelength. The angle-corrected and scaled wavelength may then be utilized to create, generate, and / or select a Gabor filter or a family of directional Gabor filters that may be applied to the image data, as described above. Examples of angle correction factors include or are the cosine of the angle at which the longitudinal axis of the object 302 is oriented relative to the imaging plane.

[0033] As another illustrative, non-exclusive example, camera system 308 may rotate about object 302, about the position of object 302, and / or about the longitudinal axis of object 302. Such rotation of camera system 308 causes a rotation of the image plane, thereby providing three-dimensional information about object 302 and allowing system 300 to utilize a family of directional Gabor filters.

[0034] In some methods 100 and systems 300, the filtered data includes or can be described as one or more blobs of pixels that are candidates for representing an object 302. Some such methods 100 also include, after applying (104) the filter, analyzing (120) the one or more blobs of pixels (i.e., local groupings) to determine the presence of one or more blob features. In the corresponding system 300, the computer-readable instructions, when executed, thus further cause the controller to analyze the one or more blobs of pixels to determine the presence of the one or more blob features after applying the filter. For example, the one or more blob features may include one or more of blob area, blob eccentricity, blob size, blob brightness, blob correlation, and blob homogeneity. In some such methods 100 and systems 300, the one or more blob features are associated with one or more predetermined features 306 in a database associated with the working area 304. As discussed, the product being assembled or manufactured may have a parts list stored in a database in which features of the parts are identified. Thus, if an object in working area 304 is expected or desired to have assumed characteristics corresponding to the expected area, eccentricity, size, brightness, correlation, and / or homogeneity of one or more spots, analysis of the one or more spots helps to ultimately determine whether object 302 has one or more predetermined characteristics.

[0035] Some such methods 100 also include training 122 a machine learning model to identify one or more predetermined features associated with the one or more blob features 306. In the corresponding system 300, the computer readable instructions, when executed, thus further cause the controller to train the machine learning model to identify one or more predetermined features associated with the one or more blob features 306. As an illustrative, non-exclusive example, the machine learning model may be a support vector method (SVM).

[0036] In some methods 100 and systems 300, the camera system 308 is a stereo camera system, and thus the image data includes two images. Such methods 100 also include creating (124) a point cloud of filtered data. In corresponding systems 300, the computer readable instructions, when executed, thus further cause the controller 310 to create a point cloud of filtered data.

[0037] Some such methods 100 also include projecting 126 a light texture on the work area 304 during capturing 102 image data. The corresponding system 300 also includes a projector 316 configured to project the light texture on the work area 304. In some such methods 100 and systems 300, the light texture includes a pattern, while in other such methods 100 and systems 300, the light texture is random. By projecting the light texture on the work area 304, two images captured by the camera system 308 can be resolved to create a point cloud. For example, in some methods 100, creating (124) the point cloud can include generating 128 a disparity map from the two images based on the light texture.

[0038] More specifically, some methods 100 also include selecting (130) pixels associated with object 302 and comparing (132) the pixels associated with object 302 to a computer model (e.g., a CAD model) of an expected or desired object from a database associated with work area 304. In a corresponding system 300, the computer-readable instructions, when executed, thus further cause a controller to select pixels associated with object 302 and compare the pixels associated with object 302 to the computer model of the expected or desired object from a database associated with work area 304. As an illustrative, non-exclusive example, the computer model of the fastener may include a representation of a cylinder corresponding to the fastener axis, and comparing the computer model to the pixels associated with object 302 may include fitting the cylinder to the pixels using normal and radius estimates.

[0039] In some applications, the work area 304 includes a plurality of objects 302. Thus, some methods 100 are performed in conjunction with each of the plurality of objects. In the corresponding system 300, the computer readable instructions, when executed, thus further cause the controller 310 to determine whether each of the plurality of objects 302 has a particular characteristic.

[0040] Continue to refer Figure 1 and 2 , the robot installation method 200 and the robot installation system 400 are also within the scope of the present disclosure. Figure 1 Schematically shown and generally referred to in Figure 2 , the robotic installation method 200 includes performing the automated machine vision method 100 according to the present disclosure, wherein the camera system 308 is mounted to, with, or as an end effector of a robotic arm 402, and based on the determination (106) of the performed method 100, instructing (202) the robotic arm 402 to install the component in a predetermined configuration relative to the object 302, and using the robotic arm 402 to install (204) the component in the predetermined configuration relative to the object 302. Figure 2, the robotic mounting system 400 includes a machine vision system 300 and a robotic arm 402 according to the present disclosure, in which the camera system 308 is mounted to, with, or as an end effector of the robotic arm 402, and wherein the robotic arm 402 is configured to mount the component in a predetermined configuration relative to the object 302 based in part on a determination by the controller 310 of whether the object 302 has a particular feature and instructions received from the controller 310. That is, the computer-readable instructions, when executed, also cause the controller to instruct the robotic arm to mount the component in the predetermined configuration relative to the object.

[0041] Now go to Figure 3 , shows an illustrative, non-exclusive example of a robotic mounting system 400 in the form of a robotic mounting system 401. Where appropriate, from Figure 2 The reference numerals of the schematic diagram are used to indicate corresponding parts of the robot mounting system 401; however, Figure 3 The examples are non-exclusive and do not limit the robotic mounting system 400 to Figure 3 That is, the robot mounting system 400 is not limited to the specific embodiment of the robot mounting system 401 shown, and the robot mounting system 400 can be combined with Figure 2 and / or Figure 3 Any number of the various aspects, configurations, features, characteristics, etc. of the robotic mounting system 400 shown and discussed in the schematic representations of embodiments and variations thereof are not necessarily included in the present disclosure without necessarily including all such aspects, configurations, features, characteristics, etc. For the sake of brevity, each component, part, portion, aspect, area, etc., or variation thereof, previously discussed may not be discussed, illustrated, and / or labeled with respect to the robotic mounting system 401; however, the previously discussed features, variations, etc. may be used with the robotic mounting system 401 within the scope of the present disclosure.

[0042] like Figure 3As shown, the robotic mounting system 401 includes a robotic arm 402 having a camera system 308 as an end effector 404, and a nut runner 406. The robotic arm 402 is positioned relative to a work area 304, and the work area 304 includes a plurality of objects 302 in the form of fasteners extending from a sheet 414. More specifically, in the depicted example, a plurality of bolts 408 extend through the sheet 414, wherein a portion of the bolts 408 have nuts 410 mounted thereon and define fastener pairs 412. The controller 310 is schematically presented as communicating with the camera system 308, the nut runner 406, and the robotic arm 402, the operational communication being schematically represented by lightning, and the communication may be wired and / or wireless. Thus, the robotic mounting system 401 is configured to perform methods 100 and 200 according to the present disclosure. More specifically, the robotic mounting system 401 utilizes its machine vision system 300 to determine whether the object 302 has one or more predetermined features, such as whether the correct bolt 408 and nut 410 are present, whether the bolt 408 is correctly or fully positioned to operably receive the nut 410, whether the nut is correctly and operably positioned on the corresponding bolt 408, whether the bolt 408 still requires the nut 410, etc. Then, based on such determinations, the robotic arm 402 may utilize the nut runner 406 to operably install additional nuts 410, and / or may not install nuts 410, for example, where an incorrect bolt 408 is positioned, or where the bolt 408 is not operably positioned to receive the nut 410.

[0043] Now go to Figure 4 and Figure 5 , the embodiments of the present disclosure may be used in Figure 4 Aircraft manufacturing and service method 500 and the like are shown Figure 5 That is, the method 500 may include one or more of the methods 100 or 200 according to the present disclosure, and the aircraft 600 may be manufactured or maintained using the system 300 or 400 according to the present disclosure, for example where the aircraft 600 includes the work area 304.

[0044] During pre-production, exemplary method 500 may include specification and design 504 of aircraft 600 and material procurement 506. During production, component and subassembly manufacturing 508 and system integration 510 of aircraft 600 may occur. Thereafter, aircraft 600 may be certified and delivered 512 for entry into service 514. During customer service, aircraft 600 may be scheduled for routine maintenance and service 516 (which may also include modification, reconfiguration, refurbishment, etc.).

[0045] Each process of method 500 may be performed or executed by a system integrator, a third party, and / or an operator (e.g., a customer). For the purposes of this specification, a system integrator may include, but is not limited to, any number of aircraft manufacturers and major system subcontractors; a third party may include, but is not limited to, any number of suppliers, subcontractors, and vendors; and an operator may be an airline, a leasing company, a military entity, a service organization, etc.

[0046] like Figure 5 As shown, an aircraft 600 produced by exemplary method 500 may include a fuselage 602 having a plurality of systems 604 and an interior cabin 606. Examples of high-level systems 604 include one or more of a propulsion system 608, an electrical system 610, a hydraulic system 612, and an environmental system 614. Any number of other systems may also be included. Although an aerospace example is shown, the principles of the invention disclosed herein may be applied to other industries, such as the automotive industry.

[0047] The apparatus and methods disclosed herein may be employed during any one or more stages of the production and service method 500. For example, components and subassemblies corresponding to the production stage 508 may be manufactured or fabricated in a manner similar to components and subassemblies produced while the aircraft 600 is in service. Furthermore, during the production stages 508 and 510, one or more apparatus embodiments, method embodiments, or a combination thereof may be utilized, for example, by significantly speeding up assembly of the aircraft 600 or reducing the cost of the aircraft 600. Similarly, one or more apparatus embodiments, method embodiments, or a combination thereof may be utilized while the aircraft 600 is in service, such as, but not limited to, maintenance and service 516.

[0048] Illustrative, non-exclusive examples of the inventive subject matter according to the present disclosure are described in the following enumerated paragraphs:

[0049] A. An automated machine vision method for determining whether an object in a work area has one or more predetermined features, the automated machine vision method comprising:

[0050] capturing image data of the work area from a camera system;

[0051] applying a filter to the image data to produce filtered data, wherein the filter includes an aspect corresponding to a particular characteristic of the one or more predetermined characteristics; and

[0052] Based at least in part on the application, a determination is made as to whether the object has a particular characteristic.

[0053] A1. The automated machine vision method of paragraph A, wherein at least one of:

[0054] (i) positioning the camera system at a known position and orientation relative to the work area; and

[0055] (ii) The automated machine vision method also includes determining a relative orientation between the camera system and the work area.

[0056] A2. The automated machine vision method of any of paragraphs A-A1, wherein the camera system is mounted to, mounted with, or mounted as an end effector of a robotic arm.

[0057] A3. The automatic machine vision method of any of paragraphs A-A2, wherein the image data includes color data.

[0058] A4. The automated machine vision method of any of paragraphs A-A3, further comprising:

[0059] Illuminates the work area during capture.

[0060] A5. The automated machine vision method of any of paragraphs A-A4, further comprising, after capturing the image data and before applying the filter:

[0061] Based on the known color of the portion of the working area that is not the object, data corresponding to the portion of the working area that is not the object is subtracted from the image data.

[0062] A6. The automated machine vision method of any of paragraphs A-A5, further comprising, after capturing the image data and before applying the filter:

[0063] Convert image data to HSV domain.

[0064] A7. The automated machine vision method of any of paragraphs A-A6, further comprising, after capturing the image data and before applying the filter:

[0065] Binary thresholding the image data based on hypothesized features of one or more predetermined features; and

[0066] In response to the binary thresholding, one or more groups of pixels are identified as candidates for putative features.

[0067] A7.1. The automatic machine vision method of paragraph A7, wherein the feature is assumed to correspond to an expected or desired object within the work area based on a database associated with the work area.

[0068] A7.2. The automated machine vision method of any of paragraphs A7-A7.1, wherein the feature is assumed to be one or more of color, shape, size, marking, and thread configuration.

[0069] A7.3. The automated machine vision method of any of paragraphs A7-A7.2, further comprising:

[0070] Noise filtering of image data after binary thresholding;

[0071] Wherein identifying one or more groups of pixels is further responsive to noise filtering.

[0072] A8. The automatic machine vision method of any of paragraphs A-A7.3, wherein the specific feature is a presumed texture of one or more predetermined features.

[0073] A8.1. The automatic machine vision method of paragraph A8, wherein the texture is assumed to correspond to an expected or desired object in the working area based on a database associated with the working area.

[0074] A8.2. The automated machine vision method of any of paragraphs A8-A8.1, wherein the texture is assumed to correspond to a fastener thread size.

[0075] A9. The automatic machine vision method of any of paragraphs A-A8.2, wherein the filter is a Gabor filter.

[0076] A10. The automatic machine vision method of any of paragraphs A-A9, wherein the filtered data includes one or more pixel blobs that are candidates for representing an object.

[0077] A10.1. The automated machine vision method of paragraph A10, further comprising:

[0078] After applying the filter, the one or more pixel blobs are analyzed to determine the presence of one or more blob features.

[0079] A10.1.1. The automatic machine vision method of paragraph A10.1, wherein the one or more blob characteristics may include one or more of blob area, blob eccentricity, blob size, blob brightness, blob correlation, and blob homogeneity.

[0080] A10.1.1.1. The automated machine vision method of paragraph A10.1.1, wherein the one or more blob features are associated with one or more predetermined features in a database associated with the work area.

[0081] A10.1.2. The automated machine vision method of any of paragraphs A10-A10.1.1.1, further comprising:

[0082] The machine learning model is trained to identify one or more predetermined features associated with the one or more blob features.

[0083] A11. The automated machine vision method of any of paragraphs A-A10.1.2, wherein the camera system is a stereo camera system, wherein the image data includes two images, and wherein the automated machine vision method further comprises:

[0084] Create a point cloud of filtered data.

[0085] A11.1. The automated machine vision method of paragraph A11, further comprising:

[0086] During the capture of image data, a light texture is projected onto the work area.

[0087] A11.1.1. The automatic machine vision method of paragraph A11.1, wherein the light texture comprises a pattern.

[0088] A11.1.2. The automated machine vision method of paragraph A11.1, wherein the light texture is random.

[0089] A11.1.3. The automated machine vision method of any of paragraphs A11.1-A11.1.2, wherein creating the point cloud comprises generating a disparity map from two images based on light texture.

[0090] A11.1.3.1 The automated machine vision method of paragraph A11.1.3, further comprising:

[0091] selecting pixels associated with the object; and

[0092] Pixels associated with the object are compared to a computer model of an expected or desired object from a database associated with the working area.

[0093] A12. The automated machine vision method of any of paragraphs A-A11.1.3.1, wherein the object comprises a fastener or a pair of fasteners.

[0094] A13. The automated machine vision method of any of paragraphs A-A12, wherein the work area includes a plurality of objects, and wherein the method is performed in conjunction with each of the plurality of objects.

[0095] A14. The automated machine vision method of any of paragraphs A-A13, wherein the work area includes a plurality of objects extending from the sheet.

[0096] A14.1. The automated machine vision method of paragraph A14, when dependent on paragraph A5, wherein the portion of the working area that is not an object corresponds to a sheet of material.

[0097] A15. The automatic machine vision method of any of paragraphs A-A14.1, wherein the one or more predetermined features include one or more of color, size, shape, marking, and thread configuration.

[0098] A16. The automatic machine vision method of any of paragraphs A-A15 is performed by the machine vision system of any of paragraphs C-C15.

[0099] B. A robot installation method, comprising:

[0100] The automated machine vision method of any of paragraphs A-A16, wherein the camera system is mounted to, with, or as an end effector of a robotic arm;

[0101] Based on the determination, instructing the robotic arm to install the component in a predetermined configuration relative to the object; and

[0102] A robotic arm is used to mount components in a predetermined configuration relative to an object.

[0103] B1. The robotic installation method of paragraph B, wherein the object and the assembly include fastener pairs.

[0104] C. A machine vision system for determining whether an object in a work area has one or more predetermined features, the machine vision system comprising:

[0105] a camera system configured to capture image data of the work area; and

[0106] a controller communicatively coupled to the camera system to receive image data from the camera system, wherein the controller includes a non-transitory computer readable medium having computer readable instructions that, when executed, cause the controller to:

[0107] applying a filter to the image data to create filtered data, wherein the filter includes aspects corresponding to particular features of the one or more predetermined features; and

[0108] Based at least in part on the application of the filter, a determination is made as to whether the object has a particular characteristic.

[0109] C1. The machine vision system of paragraph C, further comprising:

[0110] a camera stand, wherein the camera system is supported by the camera stand;

[0111] Wherein, the controller is communicatively coupled to the camera support to receive position and orientation data of the camera system relative to the working area.

[0112] C1.1. The machine vision system of paragraph C1, wherein the camera support is a robotic arm.

[0113] C2. The machine vision system of any of paragraphs C-C1.1, wherein the image data includes color data.

[0114] C3. The machine vision system of any of paragraphs C-C2, further comprising:

[0115] A lighting device configured to illuminate the work area while the camera system captures image data.

[0116] C4. The machine vision system of any of paragraphs C-C3, wherein the computer readable instructions, when executed, further cause the controller to:

[0117] Prior to applying the filter, data corresponding to portions of the working area that are not objects are subtracted from the image data based on the known colors of the portions of the working area that are not objects.

[0118] C5. The machine vision system of any of paragraphs C-C4, wherein the computer readable instructions, when executed, further cause the controller to:

[0119] Before applying the filter, convert the image data to the HSV domain.

[0120] C6. The machine vision system of any of paragraphs C-C5, wherein the computer readable instructions, when executed, further cause the controller to:

[0121] Prior to applying the filter, binary thresholding the image data based on hypothesized characteristics of one or more predetermined characteristics; and

[0122] Responsive to the binary thresholded image data, one or more groups of pixels are identified as candidates for putative features.

[0123] C6.1. The machine vision system of paragraph C6, wherein the feature is assumed to correspond to an expected or desired object in the work area based on a database associated with the work area.

[0124] C6.2. The machine vision system of any of paragraphs C6-C6.1, wherein the assumed feature is one or more of color, shape, size, marking, and thread configuration.

[0125] C6.3. The machine vision system of any of paragraphs C6-C6.2, wherein the computer-readable instructions, when executed, further cause the controller to:

[0126] After the image data is binary thresholded, noise filtering the image data;

[0127] Therein, identifying one or more groups of pixels is further responsive to the noise filtered image data.

[0128] C7. The machine vision system of any of paragraphs C-C6.3, wherein the specific feature is an assumed texture of one or more predetermined features.

[0129] C7.1. The machine vision system of paragraph C7, wherein the texture is assumed to correspond to an expected or desired object in the work area based on a database associated with the work area.

[0130] C7.2. The machine vision system of any of paragraphs C7-C7.1, wherein the texture is assumed to correspond to a fastener thread size.

[0131] C8. A machine vision system of any of paragraphs C-C7.2, wherein the filter is a Gabor filter.

[0132] C9. A machine vision system of any of paragraphs C-C8, wherein the filtered data includes one or more pixel blobs that are candidates for representing an object.

[0133] C9.1. The machine vision system of paragraph C9, wherein the computer readable instructions, when executed, further cause the controller to:

[0134] After applying the filter, the one or more pixel blobs are analyzed to determine the presence of one or more blob features.

[0135] C9.1.1. The machine vision system of paragraph C9.1, wherein the one or more speckle characteristics may include one or more of spot area, spot eccentricity, spot size, spot brightness, spot correlation, and spot homogeneity.

[0136] C9.1.1.1. The machine vision system of paragraph C9.1.1, wherein the one or more blob features are associated with one or more predetermined features in a database associated with the work area.

[0137] C9.1.2. The machine vision system of any of paragraphs C9-C9.1.1.1, wherein the computer-readable instructions, when executed, further cause the controller to:

[0138] The machine learning model is trained to identify one or more predetermined features associated with the one or more blob features.

[0139] C10. The machine vision system of any of paragraphs C-C9.1.2, wherein the camera system is a stereo camera system, wherein the image data includes two images, and wherein the computer-readable instructions, when executed, further cause the controller to:

[0140] Create a point cloud of filtered data.

[0141] C10.1. The machine vision system of paragraph C10, further comprising:

[0142] A projector configured to project a light pattern onto a work area.

[0143] C10.1.1. The machine vision system of paragraph C10.1, wherein the light texture comprises a pattern.

[0144] C10.1.2. The machine vision system of paragraph C10.1, wherein the light texture is random.

[0145] C10.1.3. The machine vision system of any of paragraphs C10.1-C10.1.2, wherein creating the point cloud includes generating a disparity map from two images based on light texture.

[0146] C10.1.3.1 The machine vision system of paragraph C10.1.3, wherein the computer readable instructions, when executed, further cause the controller to:

[0147] selecting pixels associated with the object; and

[0148] Pixels associated with the object are compared to a computer model of an expected or desired object from a database associated with the working area.

[0149] C11. The machine vision system of any of paragraphs C-C10.1.3.1, wherein the object comprises a fastener or a pair of fasteners.

[0150] C12. The machine vision system of any of paragraphs C-C11, wherein the work area includes a plurality of objects, and wherein the computer-readable instructions, when executed, further cause the controller to determine whether each of the plurality of objects has a particular feature.

[0151] C13. The machine vision system of any of paragraphs C-C12, wherein the work area includes a plurality of objects extending from the sheet.

[0152] C13.1. The machine vision system of paragraph C13, when dependent on paragraph C4, wherein the portion of the working area that is not an object corresponds to a sheet of material.

[0153] C14. The machine vision system of any of paragraphs C-C13.1, wherein the one or more predetermined characteristics include one or more of color, size, shape, marking, and thread configuration.

[0154] C15. The machine vision system of any of paragraphs C-C14, configured to perform the automatic machine vision method of any of paragraphs A-A16.

[0155] C16. Using the machine vision system of any of paragraphs C-C15 to determine whether an object within a work area has one or more predetermined features.

[0156] D. A robot installation system, comprising:

[0157] A machine vision system as described in any of paragraphs C-C15; and

[0158] a robotic arm, wherein the camera system is mounted to, with, or as an end effector of the robotic arm, and wherein the robotic arm is configured to mount the component in a predetermined configuration relative to the object based in part on a determination by the controller that the object has a particular feature;

[0159] Wherein the computer readable instructions, when executed, further cause the controller to instruct the robotic arm to install the component in a predetermined configuration relative to the object.

[0160] D1. The robotic mounting system of paragraph D, wherein the object and the assembly include fastener pairs.

[0161] D2. Using the robotic mounting system of any of paragraphs D-D1 to mount a component in a predetermined configuration relative to an object.

[0162] As used herein, the terms "adaptation" and "configuration" mean that an element, component or other subject is designed and / or intended to perform a given function. Therefore, the use of the terms "adaptation" and "configuration" should not be interpreted as meaning that a given element, component or other subject is only "able to" perform a given function, but rather that the element, component and / or other subject is specifically selected, created, implemented, utilized, programmed and / or designed in order to perform the function. Also within the scope of the present disclosure, elements, components and / or other cited subjects cited as being suitable for performing a specific function may be additionally or alternatively described as being configured to perform the function, and vice versa. Similarly, subjects cited as being configured to perform a specific function may be additionally or alternatively described as being operable to perform the function.

[0163] As used herein, the term "and / or" placed between a first entity and a second entity means one of (1) the first entity, (2) the second entity, and (3) the first entity and the second entity. Multiple items listed with "and / or" should be interpreted in the same manner, namely, "one or more" entities so combined. In addition to the entities specifically identified by the "and / or" terminology, other entities may optionally be present, whether or not related to those specifically identified entities. Thus, as a non-limiting example, in one example, when used in conjunction with an open language such as "includes," a reference to "A and / or B" may refer to only A (optionally including entities other than B); in another example, to only B (optionally including entities other than A); and in yet another example, to both A and B (optionally including other entities). These entities may refer to elements, actions, structures, steps, operations, values, etc.

[0164] According to all devices and methods disclosed herein, the steps of various disclosed elements and methods of the device disclosed herein are not required, and the disclosure includes all novel and non-obvious combinations and sub-combinations of various elements and steps disclosed herein. In addition, one or more of the various elements and steps disclosed herein can define independent invention themes, which are independent of the entire disclosed device or method. Therefore, such invention themes do not need to be associated with the specific devices and methods clearly disclosed here, and such invention themes can find uses in devices and / or methods not clearly disclosed herein.

Claims

1. An automated machine vision method (100) for determining whether an object (302) in a work area (304) has one or more predetermined features (306), wherein: The object is a threaded fastener, and the automatic machine vision method (100) comprises: capturing (102) image data of the working area (304) from a camera system (308); After capturing (102) the image data, binary thresholding (114) the image data based on assumed features of the one or more predetermined features (306), wherein the assumed features are one or more of color, shape, size, markings, and thread configuration; Responsive to the binary thresholding (114), identifying (116) one or more groups of pixels as candidates for the putative feature; After identifying (116) the one or more groups of pixels, applying (104) a Gabor filter to the one or more groups of pixels to generate filtered data, wherein the filter includes an aspect corresponding to a particular feature of the one or more predetermined features (306), wherein the particular feature is a particular pitch; and Based at least in part on the application (104), it is determined (106) whether the object (302) has the particular characteristic.

2. The automatic machine vision method (100) according to claim 1, wherein: At least one of: (i) positioning the camera system (308) at a known position and orientation relative to the work area (304); and (ii) The automated machine vision method (100) further includes determining (106) a relative position between the camera system (308) and the working area (304).

3. The automated machine vision method (100) of claim 1, further comprising, after capturing (102) the image data and before applying (104) the filter: converting (112) the image data into the HSV domain; and Based on the known color of the portion of the working area (304) that is not the object (302), data corresponding to the portion of the working area (304) that is not the object (302) is subtracted (110) from the image data.

4. The automatic machine vision method (100) according to claim 1, wherein: The assumed features correspond to expected or desired objects in the work area (304) based on a database associated with the work area (304).

5. The automatic machine vision method (100) according to claim 1, further comprising: Noise filtering (118) the image data after the binary thresholding (114); wherein the identifying (116) one or more groups of pixels is also responsive to the noise filtering (118).

6. The automatic machine vision method (100) according to claim 1, wherein: The filtered data includes one or more pixel blobs that are candidates for representing the object (302), and wherein the automated machine vision method (100) further comprises: After applying (104) the filter, the one or more pixel blobs are analyzed (120) to determine the presence of one or more blob features, wherein the one or more blob features include one or more of blob area, blob eccentricity, blob size, blob brightness, blob correlation, and blob homogeneity, and wherein the one or more blob features are associated with the one or more predetermined features (306) in a database associated with the working area (304).

7. The automatic machine vision method (100) of claim 1, wherein: The camera system (308) is a stereo camera system, wherein the image data comprises two images, and wherein the automatic machine vision method (100) further comprises: During the capturing (102) of the image data, projecting (126) a light texture onto the working area (304); creating (124) a point cloud of filtered data, wherein said creating (124) the point cloud comprises generating (128) a disparity map from said two images based on said light texture; selecting (130) pixels associated with the object (302); and The pixels associated with the object (302) are compared (132) to a computer model of an expected or desired object from a database associated with the working area (304).

8. A machine vision system (300) for determining whether an object (302) in a work area (304) has one or more predetermined features (306), wherein: The object is a threaded fastener, and the machine vision system (300) comprises: a camera system (308) configured to capture image data in the working area (304); and a controller (310) communicatively coupled to the camera system (308) to receive the image data from the camera system (308), wherein the controller (310) includes a non-transitory computer-readable medium having computer-readable instructions that, when executed, cause the controller (310) to: binary thresholding the image data based on a presumed characteristic of the one or more predetermined characteristics (306), wherein the presumed characteristic is one or more of color, shape, size, marking, and thread configuration; responsive to the binary thresholding, identifying one or more groups of pixels as candidates for the putative feature; After identifying the one or more groups of pixels, applying a Gabor filter to the one or more groups of pixels to produce filtered data, wherein the filter includes an aspect corresponding to a particular feature of the one or more predetermined features (306), wherein the particular feature is a particular pitch; and Based at least in part on the application of the filter, a determination is made as to whether the object (302) has the particular characteristic.

Images