Control system and method for a parts assembly workstation

Abstract

A method for controlling a parts assembly workstation (100) for assembling parts (102) includes imaging an assembly area (110) of the parts assembly workstation using at least one imaging device (150) to create a video of the assembly area during assembly. The method defines motion regions (312, 314) within the video. The motion regions are subsets of the assembly areas. The method detects objects within the assembly areas of the video and segments the video into motion clips based on the positions of the detected objects relative to the defined motion regions.

Description

Technical Field

[0001] The main topic of this article is parts assembly workstations. Background Technology

[0002] In many manufacturing environments, complex parts are assembled manually or through automated assembly processes. Parts may be assembled at different workstations by different operators performing the same or different assembly tasks. Some systems use vision systems to capture video data of the assembly process, which is then analyzed to optimize the manufacturing process or compare the efficiency of different operators. The video data is typically segmented into relevant clips corresponding to specific stages of the assembly. This segmentation process is usually performed manually by engineers who review the footage and extract meaningful segments for further inspection. This manual approach is both time-consuming and susceptible to operator bias, as it relies on human judgment to accurately identify critical events. Due to variations in perception, fatigue, and supervision, this process can lead to inconsistencies, resulting in inefficiencies in diagnosing assembly problems and implementing corrective actions.

[0003] The increasing complexity of manufacturing workflows and the growth in video data volume have exacerbated these challenges. The need for faster, more accurate, and automated video segmentation and part assembly methods is crucial for improving efficiency, reducing errors, and enhancing overall quality control. Summary of the Invention

[0004] In one embodiment, a method is provided for controlling a part assembly workstation for assembling parts. The method includes imaging an assembly area of ​​the part assembly workstation using at least one imaging device to create a video of the assembly area during an assembly process. The method defines a region of motion in the video. The region of motion is a subset of the assembly area. The method detects objects within the assembly area of ​​the video and segments the video into motion clips based on the positions of the detected objects relative to the defined region of motion. Attached Figure Description

[0005] The accompanying drawings will be described by way of example with reference to the drawings, wherein:

[0006] Figure 1 A parts assembly workstation 100 according to an exemplary embodiment is shown.

[0007] Figure 2 A control system 200 for a parts assembly workstation 100 according to an exemplary embodiment is shown.

[0008] Figure 3 A control system 200 for a parts assembly workstation 100 according to an exemplary embodiment is shown.

[0009] Figure 4This is a flowchart illustrating a method 400 for controlling a parts assembly workstation according to an exemplary embodiment for assembling parts. Detailed Implementation

[0010] The embodiments described herein relate to video-based analytics for video segmentation of manual assembly actions performed at one or more part assembly workstations. The control system that controls the operation of the part assembly workstation performs the video-based analytics, and artificial intelligence (AI) techniques can be used to assist video segmentation. Assembly process analysis helps optimize operations by understanding operator behavior patterns and action cycle times. Assembly process analysis may take many times longer than the video used for collecting operation cycle time data. Assembly process analysis and / or video segmentation can benefit manufacturers by interpreting human actions within the production environment. For example, video segmentation can be used for workstation training or rebalancing to improve overall assembly line throughput. Video segmentation can be analyzed to determine the timing of actions or tasks. Video segmentation can be analyzed to compare operator efficiency. Video segmentation can be analyzed to identify various actions and / or wasteful or inefficient steps in the assembly process.

[0011] The various embodiments described herein include an AI-based region-triggered method for manual assembly action segmentation to assist engineers in manufacturing process analysis. Automated assembly action segmentation can significantly reduce the time spent on cycle-time data collection. Automated assembly action segmentation can divide manual assembly operation video into different video clips, each representing a different action. Using these video clips, engineers can analyze the action clips in detail to optimize the manufacturing process.

[0012] The various embodiments described herein may include (i) an object detection model that detects and locates key objects (e.g., an operator’s left and right hands and / or parts and / or tools used in an assembly) in video frames, and / or (ii) a video analysis tool that allows a user to define a region of motion to determine when the motion begins and ends, and / or (iii) a control algorithm that segments the video into different motion clips based on the defined region and the location of the detected objects.

[0013] Figure 1A parts assembly workstation 100 according to an exemplary embodiment is shown. The parts assembly workstation 100 includes an assembly area 110 in which parts 102 are assembled to produce a product 104. Parts 102 and / or product 104 may be referred to as workpieces or objects. The parts assembly workstation 100 can be used to manufacture various types of products, such as electrical connectors, medical devices, or other types of products. The parts assembly workstation 100 includes a control system 200 for controlling the operation of the parts assembly workstation 100. In an exemplary embodiment, the control system 200 provides video-based analysis of the parts assembly process at the parts assembly workstation 100.

[0014] In an exemplary embodiment, the parts assembly workstation 100 includes a support structure 112 at an assembly area 110 for supporting other components. The support structure 112 may include a workbench, frame, fixing device, assembly mandrel, or other support structure. Parts may be assembled at the assembly area 110 above the support structure 112. In various embodiments, the parts assembly workstation 100 includes a retainer 114 for holding one or more of parts 102 or products 104 during assembly. The retainer 114 may include a fixing device or mandrel for holding part 102 in a fixed position within the assembly area 110, for example, for assembling other parts to part 102. In various embodiments, the parts assembly workstation 100 includes an assembly tool 116 for assisting in the assembly of part 102. The assembly tool 116 can be used to insert one part into another part. The assembly tool 116 can be used to secure or fasten parts together, for example, for crimping, brazing, welding, or otherwise securing parts. The assembly tool 116 can be used to form one or more parts. Other types of assembly tools may be used in alternative embodiments.

[0015] In various embodiments, the parts assembly workstation 100 includes a parts storage section 120 for storing parts 102. For example, the parts storage section 120 may include a storage box, storage tray, storage reel, or other type of parts storage component. In various embodiments, the parts assembly workstation 100 includes a product storage section 122 for storing products 104 after assembly. For example, the product storage section 122 may include a box, tray, reel, or other type of product storage component.

[0016] In an exemplary embodiment, the parts assembly workstation 100 includes one or more imaging devices 150 at an assembly area 110. The imaging devices 150 may be mounted to a support structure 112. In an exemplary embodiment, the imaging device 150 includes a camera 152. The imaging device 150 generates video of the assembly area 110 during the assembly process for analysis, such as to optimize the manufacturing process. The entire assembly area 110 may be the target of inspection, or only a portion of the assembly area 110 may be the target of inspection. The assembly area 110 may include multiple inspection targets, which may be imaged by one of the imaging devices 150 or by different imaging devices 150. The resulting images / videos may include multiple target objects, which can be analyzed manually or by AI / vision-based analysis to segment the video into action clips corresponding to assembly actions / tasks. The imaging device 150 may image the assembly area from multiple angles. The imaging device 150 may image multiple assembly tasks, for example, in different areas of the assembly area 110 or at different assembly areas 110. In an exemplary embodiment, the imaging device 150 creates video of the assembly area 110 during the assembly process. Imaging device 150 may be a digital camera for generating digital images. Imaging device 150 may include a machine vision camera, such as a region scan camera. Imaging device 150 may include a CCD camera, a CMOS camera, or other types of cameras. Imaging device 150 may generate monochrome or color images. In various other embodiments, imaging device 150 may include a thermal camera or other types of imaging device for generating thermal images.

[0017] In an exemplary embodiment, the control system 200 is operatively coupled to the imaging device 150. The control system 200 receives video from the imaging device 150 for analysis of the assembly process. In an exemplary embodiment, the control system 200 includes a video analysis tool 202, an object detection tool 204, and a video segmentation tool 206. The video analysis tool 202 is configured to define a region of motion in the video. The object detection tool 204 is configured to detect objects in the assembly region 110 of the video. The video segmentation tool 206 is configured to segment the video into motion clips for analysis based on the position of the detected objects relative to the defined region of motion.

[0018] In an exemplary embodiment, the system is designed for AI-based region-triggered video analysis of manual part assembly actions to assist engineers in manufacturing process analysis. The system prepares a set of video data for the target action. In various embodiments, an operator can use video analysis tool 202 to specify the action region within the video. The action region can be manually set by the operator in the video (e.g., position, size, shape, etc.). Alternatively, the action region can be automatically detected and set by performing video analysis to identify specific areas in the video where repetitive tasks occur. The action region is a region or sub-region within assembly area 110 where one or more assembly steps occur. The action region can be the area where the operator's hands are located during the assembly task. The action region can be the area where a fixture or assembly tool is located, or the area where the assembly of the part occurs.

[0019] In various embodiments, the operator can define the start and stop conditions for each action. For example, the operator can use video analysis tool 202 to define the start and stop conditions. The start and stop conditions can be defined based on the position of the operator's hands (e.g., hands can move into or out of the area to perform various tasks). In various embodiments, the action begins when both hands appear in the defined action area (e.g., at the center of the frame) and ends when one or more hands move out of the action area (e.g., the left hand moves to the top left corner).

[0020] In an exemplary embodiment, object detection tool 204 is trained to identify objects within the video to distinguish different actions. For example, object detection tool 204 may be trained to identify left and right hands and / or workpieces and / or assembly tools. Object detection tool 204 can track the movement of objects within the video. In an exemplary embodiment, the control system performs object detection analysis of the video (e.g., an object detection algorithm) to obtain the bounding box position of objects (e.g., hands) in each frame.

[0021] In an exemplary embodiment, video segmentation tool 206 performs video segmentation to divide the video into different action clips for various assembly tasks. The action clips can be based on predefined action regions and detected object locations. The action clips present different actions. The action clips can be used in various use cases, such as cycle time estimation or training action segmentation AI models.

[0022] In an exemplary embodiment, the control system 200 has several features to help the user define regions and start and stop conditions for actions. The user can determine or define the number of action regions. For example, the user can specify a single action region or multiple action regions for an action / task to start or stop. The user can determine or define object classifications, such as the type of object (e.g., hand, workpiece, tool, etc.). For example, the user can define hand classifications, such as right hand, left hand, etc., to distinguish tasks based on which hand is performing the task. The user can specify the triggering of a region as: left hand, right hand, both hands, or any hand. The user can determine or define the duration of a task to qualify as an identified task. For example, the user can specify how many frames the hand needs to remain in the action region of the trigger. The user can determine or define a width threshold. For example, the user can specify a percentage of frames that must be satisfied to trigger the region. For example, if the width is 0.8 and the number of frames is 10, then an action can be triggered as long as the number of frames satisfied is equal to or greater than 8. The user can determine or define an overlap threshold to determine the amount of overlap between an object (e.g., a hand) and the action region, to limit how much of the hand enters the region to be considered valid for a frame. Video segmentation occurs to limit action clips based on specified features.

[0023] Figure 2 A control system 200 for a parts assembly workstation 100, according to an exemplary embodiment, is illustrated. The control system 200 is configured to perform video-based analysis on video segmentation of tasks / actions within a production environment to improve the assembly process. For example, video segmentation can be used for workstation training or rebalancing to increase overall assembly line throughput. Video segmentation can be analyzed to determine the timing of actions or tasks and / or compare operator efficiency and / or identify various actions and / or identify wasteful or inefficient steps in the assembly process.

[0024] The control system 200 includes an imaging device 150, a display device 210, a personal computer 220, and a control unit 250. The personal computer 220 can be omitted. Various information and images can be displayed by using the personal computer 220 instead of the display device 210. [This is already in place.] Figure 2The diagram illustrates that the control unit 250, imaging device 150, display device 210, and personal computer 220 are depicted as separate units as an example of an embodiment of the control system 200; however, multiple components can also be combined and integrated. For example, the display device 210 can be integrated with the personal computer 220, the control unit 250 can be integrated with the imaging device 150, the control unit 250 can be integrated with the display device 210, or the control unit 250 can be integrated with the personal computer 220. The control unit 250 can be divided into multiple units, and a portion of the divided units can be incorporated into the imaging device 150, the display device 210, or the personal computer 220. Alternatively, the imaging device 150 can be divided into multiple units, and a portion of the divided units can be incorporated into another unit.

[0025] Imaging device 150 includes camera 150 and may include an illumination module (not shown). Imaging device 150 is operated to acquire an image of the target to be inspected at assembly area 110. Imaging device 150 may include a motor 154 and an imaging plate 156 that drive the imaging optics system. Motor 154 is part of the system that automatically performs focus adjustment by driving the lens of camera 152, and focus adjustment may be performed by methods such as contrast autofocus. The imaging field of view (imaging area) of camera 152 is set to include the target to be inspected. Imaging plate 156 may include one or more electrical components mounted thereon, such as processors, IC components, CMOS sensors, field-programmable gate arrays (FPGAs), digital signal processors (DSPs), etc. Components of imaging device 150 may include light-receiving elements for receiving light incident from the imaging optics system. A CMOS sensor is an imaging sensor configured to acquire color images. For example, a light-receiving element such as a CCD sensor may be used instead of a CMOS sensor. FPGAs and DSPs are used to perform image processing within imaging device 150, and signals output from the CMOS sensor are also input to the FPGA and DSP.

[0026] The control unit 250 may include a motherboard 252, a communication board 254, and may include other boards, such as a power board. One or more electrical components, such as a processor, FPGA, DSP, and memory, are mounted on the motherboard 252. Components on the motherboard 252 control the operation of various connected boards and modules. For example, the control unit may output control signals for controlling the imaging device 150, display device 210, personal computer 220, or other components. The control unit may output control signals for controlling the CMOS sensor of the imaging board 156 of the camera 152, for example, to start or stop imaging and control imaging. When imaging ends, the image signal output from the imaging device 150 is input to the control unit 250, for example, to the FPGA of the motherboard 252, processed by the FPGA and DSP of the motherboard 252, and stored in the memory of the motherboard 252. The communication board 254 outputs signals such as image data, analysis output, or other results to user interfaces such as the display device 210 and / or personal computer 220. The communication board 254 can receive input from a user interface, such as a display device 210 and / or a touch panel of a personal computer 220, a keyboard of a personal computer 220, or an input device (such as a pointing device) of a personal computer 220. Communication can be wired or wireless.

[0027] In an exemplary embodiment, the control unit 250 includes one or more storage devices, such as one or more memories 260. The memory 260 may include a hard disk drive. The memory 260 stores one or more program files (software) for enabling control and processing that can be executed by hardware. For example, the program files may be stored on a storage medium such as an optical disc, and the program files stored on the storage medium may be installed in the control unit 250. The memory 260 may store image data and other analysis outputs or results.

[0028] In an exemplary embodiment, the control unit 250 includes one or more processors 270. During operation of the control unit 250, the control unit 250 runs a program to analyze an image. For example, the control unit 250 operates a program stored in memory 260 on the processor 270. The control system 200 includes a video analysis tool 202, an object detection tool 204, and a video segmentation tool 206, which may include one or more memories 260 and one or more processors 270. The video analysis tool 202 is configured to define a region of motion in the video. The object detection tool 204 is configured to detect objects in a mounting region 110 of the video. The video segmentation tool 206 is configured to segment the video into motion clips for analysis based on the position of the detected objects relative to the defined region of motion.

[0029] The processors 270 may include computer system executable instructions, such as program modules, that are executed by the computer system. Typically, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computation can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules may reside in local and remote computer system storage media, including memory storage devices.

[0030] In an exemplary embodiment, various components, such as memory 260 and processor 270, can be communicatively coupled via a bus. The bus refers to any one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of the various bus architectures.

[0031] Control unit 250 may include various computer system readable media. This media may be any available media accessible to control unit 250, and it includes volatile and non-volatile media, removable and non-removable media. Memory 260 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Control unit 250 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system may be provided for reading from and writing to a non-removable non-volatile magnetic medium (not shown and generally referred to as a "hard disk drive"). Although not shown, a disk drive may be provided for reading from and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media). In this case, each may be connected to a bus via one or more data media interfaces. Memory 260 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the invention.

[0032] One or more programs, along with an operating system, one or more applications, other program modules, and program data, may be stored in memory 260. Each or some combination of the operating system, one or more applications, other program modules, and program data may include an implementation of a networking environment. Program modules typically perform the functions and / or methods of embodiments of the subject matter described herein.

[0033] The control unit 250 can also communicate with one or more external devices, for example, via its communication board 254. External devices may include a keyboard, pointing device, display, etc.; one or more devices that enable a user to interact with the system; and / or any device that enables the system to communicate with one or more other computing devices (e.g., a network interface card, modem, etc.). This communication may occur via an input / output (I / O) interface. Furthermore, the control unit 250 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via a network adapter. Other hardware and / or software components may be used in conjunction with the system components shown herein. Examples include, but are not limited to: microcode, device drivers, redundant processing units and external disk drive arrays, RAID systems, tape drives, and data archiving storage systems.

[0034] As used herein, the term "processor" is intended to include any processing device, such as a processing device including, for example, a CPU (Central Processing Unit) and / or other forms of processing circuitry. Furthermore, the term "processor" may refer to more than one single processor. The term "memory" is intended to include memory associated with a processor or CPU, such as, for example, RAM (Random Access Memory), ROM (Read-Only Memory), fixed storage devices (e.g., hard disk drives), removable storage devices (e.g., floppy disks), flash memory, etc. Additionally, as used herein, the phrase "input / output interface" is intended to refer to interfaces, for example, for inputting data to the processing unit (e.g., a mouse) and for providing results associated with the processing unit (e.g., a printer). Processor 270, memory 260, and input / output interfaces may be interconnected, for example, via a bus, as part of a data processing unit. Suitable interconnections, such as via a bus, may also be provided to a network interface, such as a network card, which may be provided to interface with a computer network, and to a media interface, such as a disk or CD-ROM drive, which may be provided to interface with a suitable media.

[0035] Therefore, computer software including instructions or code for performing the methods described herein can be stored in one or more associated memory devices (e.g., ROM, fixed or removable memory) and partially or fully loaded (e.g., loaded into RAM) and implemented by the CPU when ready for use. Such software can include, but is not limited to, firmware, resident software, microcode, etc.

[0036] It should be noted that any method described herein may include additional steps to provide a system comprising different software modules embodied on a computer-readable storage medium; modules may include, for example, any or all suitable elements depicted in the block diagram and / or described herein; any, some, or all of the modules / blocks and / or submodules / subblocks are described by way of example and not limitation. The method steps may then be performed using the different software modules and / or submodules of a system executed on one or more hardware processors as described above. Furthermore, the computer program product may include a computer-readable storage medium having code adapted to be implemented to perform one or more method steps described herein, including providing different software modules to the system.

[0037] Figure 3 A control system 200 for a parts assembly workstation 100 according to an exemplary embodiment is shown. For example, Figure 3 A display of the control system 200 is shown, for example at display device 210. Figure 3 Multiple windows 300 on the display device 210 are shown. In an exemplary embodiment, the display device 210 shows a video player window 310 displaying video and a video player toolbar 320 for controlling the video player window 310. In an exemplary embodiment, the display device 210 shows a settings window 330 for controlling the control system 200. For example, different settings can be modified in the settings window 330 to control video-based analysis and / or video segmentation of tasks / actions within the parts assembly workstation 100. Settings can be modified in the settings window 330 to control the video analysis tool 202 and / or the object detection tool 204 and / or the video segmentation tool 206.

[0038] Video player window 310 displays video received from imaging device 150. Views from one or more cameras 152 can be displayed in video player window 310. The video shows views of assembly area 110, such as showing part 102, product 104, support structure 112, retainer 114, assembly tool 116, parts storage 120, product storage 122, or other components. The video shows an operator 130 at assembly area 110, such as the operator's right hand 132 and left hand 134, to observe the operator 130 manually assembling part 102.

[0039] In an exemplary embodiment, the video player window 310 includes one or more motion regions. For example, in the illustrated embodiment, the video player window 310 includes a first motion region 312 and a second motion region 314. Motion regions 312 and 314 may correspond to different assembly actions / tasks. For example, one or more assembly tasks may occur in the first motion region 312, and one or more assembly tasks may occur in the second motion region 314. The first motion region 312 may be an action start region, and the second motion region 314 may be an action stop region. In alternative embodiments, the video player window 310 may include more or fewer motion regions. In an exemplary embodiment, motion regions 312 and 314 are defined by bounding boxes covering the video. In alternative embodiments, other types of motion regions 312 and 314 may be provided, such as opaque or shaded regions. Motion regions 312 and 314 are regions or sub-regions within the assembly region 110, where one or more assembly steps occur. Motion regions 312 and 314 may be the areas where the operator's hands are located during an assembly task. Action areas 312 and 314 can be areas where fixing devices or assembly tools are located, or areas where the assembly of parts occurs. Action areas 312 and 314 can be defined by video analysis tool 202. For example, action areas 312 and 314 can be automatically detected and set by performing video analysis to identify specific areas in the video where repetitive tasks occur. Action areas 312 and 314 can be manually set by the operator in the video (e.g., position, size, shape, etc.). In the illustrated embodiment, action areas 312 and 314 are rectangular. However, in alternative embodiments, action areas 312 and 314 can have other shapes.

[0040] In an exemplary embodiment, the video player window 310 includes one or more object recognition regions. For example, in the illustrated embodiment, the video player window 310 includes a first object recognition region 316 and a second object recognition region 318. In alternative embodiments, the video player window 310 may include more or fewer object recognition regions. Object recognition regions 316, 318 may correspond to different objects (e.g., hands, workpieces, assembly tools, etc.). For example, the first object recognition region 316 may correspond to the operator's right hand, and the second object recognition region 318 may correspond to the operator's left hand. In an exemplary embodiment, object recognition regions 316, 318 are defined by a bounding box covering the video. In alternative embodiments, other types of object recognition regions 316, 318 may be provided, such as opaque or shaded regions. Object recognition regions can track the movement of objects within frames of the video. Object recognition regions 316, 318 may be defined by object detection tool 204. For example, object recognition regions can be automatically detected and set by performing video analysis to identify specific areas in the video where objects are located. In the illustrated embodiment, the object recognition regions are rectangular. However, in alternative embodiments, the object recognition regions may have other shapes. In an exemplary embodiment, the control system 200 determines the relative positions of the object recognition regions 316, 318 and the motion regions 312, 314 to control the system. For example, the relative positions of the object recognition regions 316, 318 and the motion regions 312, 314 are used for video segmentation. For example, when it is determined that an object exists within the motion regions 312, 314, the control system 200 associates it with a specific assembly action / task. The control system 200 segments video clips based on the overlap of the object recognition regions 316, 318 and the motion regions 312, 314.

[0041] In an exemplary embodiment, the control system 200 determines the number of motion regions. The control system 200 identifies actions / tasks to start or stop. The control system 200 determines object classifications, such as object type (e.g., hand, workpiece, tool, etc.). For example, the control system 200 may define hand classifications, such as right hand, left hand, etc., to distinguish objects in the video. The control system 200 may specify triggers for various motion regions as: left hand, right hand, both hands, or any hand. The control system 200 determines how many frames the hand is in the motion region before triggering the event. The control system 200 may use a wide threshold to trigger the event. For example, if the wide threshold is 0.8 and the number of frames is 10, then an action can be triggered as long as the number of frames is equal to or greater than 8. The control system 200 uses an overlap threshold to determine the amount of overlap between object recognition regions 316, 318 and motion regions 312, 314 to limit how much of the hand enters the region to be considered valid for a frame. Video segmentation occurs to define motion clips based on video analysis performed by the control system 200.

[0042] In the illustrated embodiment, the video player toolbar 320 is located below the video player window 310. The video player toolbar 320 includes a search bar 321 with a slider 322 indicating the position of frames within the video. In an exemplary embodiment, the video player toolbar 320 includes a visualization 324 for segmenting action clips within the video. The visualization 324 may be color-coded to identify different types of action clips. The video player toolbar 320 includes buttons 325 for controlling video playback. Buttons 325 may include one or more of the following buttons: play, pause, stop, rewind, skip, rewind, fast forward, or other buttons. Buttons 325 may include import and export buttons for importing or exporting videos or clips. Buttons 325 may include a split button for manually splitting clips from the video. The video player toolbar 320 includes identification labels 326 to identify information about the video, such as filename, frame number, action clip name, action clip start frame, action clip stop frame, or other identification information.

[0043] The settings window 330 includes settings 332 (or settings 332) that can be adjusted or modified to control system settings features. For example, settings 332 can be modified in the settings window 330 to control the video analysis tool 202 and / or the object detection tool 204 and / or the video segmentation tool 206. The operator can use settings 332 to define the action area for video segmentation and the start and stop conditions of the actions. Settings 332 may include tabs, drop-down menus, input units, buttons, toggle keys, sliders, or other types of settings. In an exemplary embodiment, settings 332 include one or more action name units 340, one or more action start duration units 342, an object selection menu 344 associated with the action start duration unit 342, one or more action stop duration units 346, an object selection menu 348 associated with the action stop duration unit 346, an action condition toggle 350, an action visibility toggle button 352, an action edit button 354, an action delete button 356, and may include other settings 332.

[0044] In an exemplary embodiment, different action names can be assigned to different types of actions / tasks performed at the part assembly workstation 100 in action name cell 340. The system can track any number of different tasks.

[0045] In an exemplary embodiment, the duration of the action start input to cell 342 can be a selected number of frames (e.g., 6 frames, 12 frames, or any other reasonable number of frames) of an object (e.g., a hand) in a corresponding action area (e.g., start action area 312). For example, the number of frames can be based on the specific task being performed and how long it takes for the operator to perform such a task. The user can select the type of object being tracked, such as right hand, left hand, part, product, assembly tool, etc., in object selection menu 344. In various embodiments, the control system can provide a selection of multiple objects in start action area 312, such as requiring both the right and left hands in start action area 312. The user can switch the required object by selecting or deselecting the action condition switch 350 associated with the various object selection menus 344. For example, the user can turn on / off the requirement of the right hand or left hand in action area 312 by selecting or deselecting the corresponding object selection menu 344.

[0046] In an exemplary embodiment, the duration of the motion stop input to unit 346 can be a selected number of frames (e.g., 6 frames, 12 frames, or any other reasonable number of frames) of an object (e.g., a hand) in a corresponding motion area (e.g., stop motion area 314). For example, the number of frames can be based on the specific task being performed and how long it takes for the operator to perform such a task. The user can select the type of object being tracked, such as right hand, left hand, part, product, assembly tool, etc., in object selection menu 348. In various embodiments, the control system can provide the option to select multiple objects in stop motion area 314, for example, requiring both the right and left hands in start motion area 314. The user can switch the required object by selecting or deselecting motion condition switching 350 associated with various object selection menus 348. For example, the user can turn on / off the requirement of the right or left hand in motion area 314 by selecting or deselecting the corresponding object selection menu 348.

[0047] In an exemplary embodiment, setting 332 includes a width threshold unit 360, where the user selects a width threshold for triggering an event. For example, if the width threshold is 0.5, an action can be triggered as long as the number of frames satisfied is equal to or greater than 0.5 times the number of frames (set at action start duration unit 342 or action stop duration unit 346). The width threshold of objects in the action region is determined in X out of Y consecutive frames.

[0048] In an exemplary embodiment, setting 332 includes an overlap threshold unit 362, wherein the user selects an overlap threshold to determine the amount of overlap between object recognition regions 316, 318 and action regions 312, 314 that is considered valid for a frame. For example, if the overlap threshold is 0.5, then a frame is considered a positive frame that triggers an action as long as 50% of the object overlaps with the action region. However, if less than 50% of the object overlaps with the action region, the frame is considered a negative frame, and therefore the frame does not trigger an action.

[0049] In an exemplary embodiment, setting 332 includes a start trigger button 364 and a stop trigger button 366. The start trigger button 364 may allow setting the number of regions used to define the start trigger condition, such as one region, two regions, three regions, or others. The start trigger button 364 may determine the type of action start trigger condition, such as the number of object types in the action region. The start trigger button 364 may require the left hand, right hand, both hands, or any hand, or it may require parts, tools, or other objects in the action region. The stop trigger button 366 may allow setting the number of regions used to define the stop trigger condition, such as one region, two regions, three regions, or others. The stop trigger button 366 may determine the type of action stop trigger condition, such as the number of object types in the action region. The stop trigger button 366 may require the left hand, right hand, both hands, or any hand, or it may require parts, tools, or other objects in the action region. The start trigger button 364 and the stop trigger button 366 may be drop-down menus.

[0050] Figure 4 This is a flowchart illustrating a method 400 for controlling a parts assembly workstation according to an exemplary embodiment for assembling parts.

[0051] At 410, the method includes imaging an assembly area of ​​a part assembly workstation using at least one imaging device to create video of the assembly area during the assembly process. Imaging may include setting up one or more cameras to image the assembly area, for example, to view one or more assembly tasks. The cameras may view the assembly area from above to image the assembly of parts into a product. The cameras may observe assembly tools, fixtures, parts storage bins, parts, assembled products, operator's hands, or other objects in the assembly area during the assembly process. The cameras may track the movement of objects within the assembly area.

[0052] At 420, the method includes defining an action region in the video. The action region is a subset of the assembly region. The action region may correspond to an area of ​​the assembly region where the assembly action of a part occurs. For example, the action region may correspond to an area where one part is inserted into or fixed to another part. In various embodiments, the step of defining the action region includes positioning the action region bounding box at a designated location in the video. For example, an operator may draw or insert one or more bounding boxes in the video at the action region. The control system may automatically determine the action region. For example, the control system may use video analysis tools to provide video analysis to determine the location of the action region in the video. The action region bounding box may have a predetermined shape, such as a rectangular shape; however, the action region bounding box is not limited to a rectangular shape. The control system may analyze the video to determine the location of the action region (e.g., based on repetitive actions or based on the location of tools, fixtures, or other components identified in the image). In various embodiments, the action region may be in a non-overlapping area of ​​the video. In other various embodiments, the action region may be an overlapping area within the video, such as when different action tasks occur in the same or similar areas.

[0053] At 430, the method includes detecting objects in an assembly area of ​​the video. The control system can automatically detect objects in the assembly area. For example, the control system can use an object detection tool to provide video analysis to determine the position of objects in the video. In various embodiments, the detected object is an operator's hand / arm. The detected object can be a workpiece (e.g., a part and / or product). The detected object can be an assembly tool, a fixture, a robotic arm, or other device used in the manufacturing process. The control system can track the movement of the object through individual frames of the video. In various embodiments, the step of detecting objects includes performing shape analysis of the video to identify objects having a predetermined shape. For example, the shape can correspond to the shape of a human hand, the shape of a workpiece, or the shape of an assembly tool. In various embodiments, the step of detecting objects includes performing color analysis of the video to identify objects having a predetermined color. For example, the color can correspond to the color of a glove worn by a person to facilitate tracking a person's hand in the video, or to the color of a part to facilitate tracking the position of the part in the video. In various embodiments, the step of detecting objects includes positioning an object bounding box in the video surrounding the object. The object bounding box can have a predetermined shape, such as a rectangular shape; however, the object bounding box is not limited to a rectangular shape.

[0054] At 440, the method includes segmenting a video into motion clips based on the position of detected objects relative to defined action regions. The segmentation of motion clips may be based on the number of action regions and the type of objects located within those regions. Segmentation may be based on the position of the action region bounding boxes and the position of the object bounding boxes. For example, segmentation may be based on the overlap between the object bounding box and the action region bounding box indicating that the object is located within the action region. The step of segmenting the video includes determining frames in the video where objects are located within action regions. In various embodiments, the segmentation step includes segmenting the video into motion clips corresponding to different actions of an assembly process. For example, the assembly process may involve multiple assembly steps, and the video may be segmented into different assembly steps. In various embodiments, the segmentation step includes determining an action start frame from the frames of the video and determining an action end frame from the frames of the video. Each motion clip may include multiple frames. For example, an action clip includes frames of the video that begin at an action start frame and end at an action end frame. In an exemplary embodiment, the action start frame is determined to be a frame where the object is located inside the action region, and the action end frame is determined to be a frame where the object is located outside the action region.

[0055] In various embodiments, the step of defining an action region includes positioning an action region bounding box at a designated location in the video, and the step of detecting an object includes positioning an object bounding box around the object in the video. The method may include the step of determining the amount of overlap between the object bounding box and the action region bounding box in each frame. The method may include the step of setting an overlap threshold for the object bounding box and the action region bounding box. In an exemplary embodiment, the segmentation step includes segmenting the video into action clips based on the overlap threshold of the object and the action region. For example, the overlap threshold may be an overlap threshold of at least 50%, wherein an object is considered to be in the action region when at least 50% of the object bounding box overlaps with the action region bounding box. The overlap threshold may be adjusted by an operator to increase or decrease the sensitivity of video segmentation. The method may include the step of setting a width threshold for the action. In an exemplary embodiment, the segmentation step includes segmenting the video into action clips based on a width threshold of objects in the action region. For example, the width threshold may correspond to the number of frames in a set of frames in which an object is identified as being in the action region. For example, in various embodiments, the width threshold may require identifying objects in the action region in X out of Y consecutive frames, where X is at least 0.5Y. The width threshold can be adjusted by the operator to increase or decrease the sensitivity of video segmentation.

[0056] At 450, the method includes providing output from the control system. The method may include outputting segmented video clips. The method may include outputting a spreadsheet showing the duration of the action clips and / or the start and end frames of the action clips and / or the frequency of occurrence of various actions / tasks in the video. The method may include visualization of the output segmentation results. For example, the visualization may be a timemap corresponding to the video. The visualization may be color-coded to show the occurrence of different actions / tasks.

Claims

1. A method for controlling a parts assembly workstation (100) for assembling parts (102), the method comprising: The assembly area (110) of the part assembly workstation is imaged using at least one imaging device (150) to create a video of the assembly area during the assembly process. The video defines motion regions (312, 314), which are subsets of the assembly regions. Detect objects in the assembly area of ​​the video; and Based on the position of the detected object relative to the defined action area, the video is segmented into action clips.

2. The method according to claim 1, wherein, The action areas (312, 314) correspond to the areas of the assembly area (110) where the assembly action of the part (102) occurs.

3. The method according to claim 1, wherein, The definition of the motion regions (312, 314) includes positioning the motion region bounding box at a specified location in the video.

4. The method according to claim 1, wherein, The assembly area (110) includes an assembly tool (116), and the definition of the action area (312, 314) includes defining an action area corresponding to the position of the assembly tool.

5. The method according to claim 1, wherein, The definition of the motion regions (312, 314) includes positioning the motion regions in the non-overlapping areas of the video.

6. The method according to claim 1, wherein, The detection target includes detecting the position of the operator's hand (134) in the video.

7. The method according to claim 1, wherein, The detection target includes shape analysis of the video for objects with predetermined shapes.

8. The method according to claim 1, wherein, The detection targets include color analysis of the video for objects with predetermined colors.

9. The method according to claim 1, wherein, The detection object includes positioning the object bounding box around the object in the video.

10. The method according to claim 1, wherein, The detection targets include objects that are tracked in each frame of the video.

11. The method according to claim 1, wherein, The segmentation of the video includes determining the frames in the video where the object is located within the action region (312, 314).

12. The method according to claim 1, wherein, The segmentation includes dividing the video into motion clips corresponding to different actions in the assembly process.

13. The method according to claim 1, wherein, The segmentation includes: determining an action start frame from the frames of the video, and determining an action end frame from the frames of the video, wherein the action clip includes frames of the video that begin at the action start frame and end at the action end frame.

14. The method according to claim 13, wherein, The action start frame is determined to be the frame in which the object is located inside the action area (312, 314), and the action end frame is determined to be the frame in which the object is located outside the action area.

15. The method according to claim 1, wherein, The definition of the action regions (312, 314) includes positioning the action region bounding box in the video at a specified location, and the detection of objects includes positioning the object bounding box in the video around the object. The method further includes determining the amount of overlap between the object bounding box and the action region bounding box in each frame.

16. The method according to claim 1, wherein, The segmentation includes dividing the video into action clips based on an overlap threshold between the object and the action regions (312, 314).

17. The method according to claim 1, wherein, The segmentation includes dividing the video into action clips based on a width threshold of objects in X of Y consecutive frames (312, 314), where X is at least 0.5Y.

18. The method of claim 1, further comprising outputting a spreadsheet showing the duration of the motion clip.

19. The method of claim 1, further comprising outputting a visualization of the segmentation result.

20. The method according to claim 1, wherein, The segmentation includes dividing the video clip into action clips based on the number of action regions (312, 314) and the type of objects located in the action regions.

21. A control system (200) for a parts assembly workstation (100), comprising: An imaging device (150) is located at the assembly area (110) of the parts assembly workstation, where parts (102) are assembled to produce a product (104), and the imaging device generates a video of the assembly area during the assembly process. A video analysis tool (202) having one or more processors (270) configured to define motion regions (312, 314) in the video, the motion regions being subsets of the assembly region; An object detection tool (204) has one or more processors configured to detect objects in the assembly region of the video; and A video segmentation tool (206) has one or more processors configured to segment a video into motion clips based on the position of a detected object relative to a defined motion region.

22. A parts assembly workstation (100), comprising: Assembly area (110), where parts (102) are assembled to produce product (104). An imaging device (150) at the assembly area, which creates a video of the assembly area during the assembly process; and A control system (200) receives the video from the imaging device, the control system including a video analysis tool (202), an object detection tool (204), and a video segmentation tool (206). The video analysis tool has one or more processors (270) configured to define motion regions (312, 314) in the video, the motion regions being subsets of the assembly region; The object detection tool has one or more processors configured to detect objects in the assembly region of the video; and The video segmentation tool has one or more processors configured to segment the video into action clips based on the position of detected objects relative to a defined action region.

Images