An industrial work pose correction method based on visual inertia

Abstract

The application discloses an industrial operation pose correction method based on visual inertia. When an industrial operation device reaches a shooting preparation position, an industrial operation image frame of an end camera, short-time attitude data of an inertial measurement unit, an operation area template, a camera and a tool end reference position relationship table and workpiece coordinate system source information are synchronously collected. Workpiece operation reference features, tool end visible edges and clamp fixed edges are extracted according to the operation area template to form an image feature state. The expected moving range of image features between adjacent image frames is limited according to the short-time attitude data. The position change, edge direction change and end point relationship change of the same image features are verified within the range to form a visual inertia continuous relationship. The candidate poses are screened in combination with the workpiece coordinate system source information, the tool end visible edge stability and the operation process tolerance to form a translation correction amount and an angle correction amount. The application reduces the correction amount jump caused by degenerated image features and improves the correction continuity.

Description

Technical Field

[0001] This invention relates to the field of industrial image processing technology, specifically to an industrial operation pose correction method based on visual inertia. Background Technology

[0002] In industrial welding, gluing, grinding, assembly positioning, and inspection sampling operations, end-effector cameras are typically mounted near the workpiece to acquire local images of the workpiece, which are then combined with inertial measurement units (IMUs) to obtain short-term attitude changes of the workpiece. This type of visual-inertial method can continuously estimate the relative pose between the image acquisition end and the workpiece's working reference during the movement of the work equipment, providing a data foundation for pose correction before operation. Existing visual-inertial pose estimation methods mostly rely on camera trajectory, moving vehicle pose, or spatial location... Figure 1 With consistency as the goal, the focus is usually on the image feature tracking, inertial pre-integration, and pose optimization results themselves.

[0003] In industrial operations, the image range acquired by the end-effector camera is relatively small. Workpiece boundaries, holes, welds, assembly reference lines, and tool edges are easily affected by metal reflections, fixture occlusion, tool end occlusion, texture repetition, and motion blur, leading to missing, mismatched, and short-term abrupt changes in image features. Although inertial measurement data can supplement motion information between adjacent image frames, inertial integration accumulates errors over time and cannot independently characterize the actual deviation of the tool end relative to the workpiece's working reference.

[0004] Therefore, when existing technologies combine degraded image features and short-term inertial predictions for pose determination, they tend to incorporate incorrect matching results into candidate pose calculations, causing discontinuous changes in translation and angle correction amounts. For industrial operation scenarios requiring precise corrections near welds, holes, assembly edges, and surface work areas, existing methods struggle to determine whether the current visual inertial pose result can serve as a valid basis for work pose correction, thus affecting the consistency and stability of subsequent work positions. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a visual-inertial-based method for industrial work posture correction, thereby solving the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for correcting the pose of industrial operations based on visual inertia, comprising the following steps: S1. When the industrial operation equipment reaches the pre-photograph position, simultaneously collect the industrial operation image frame of the end camera, the short-time attitude data of the inertial measurement unit, the operation area template, the relationship table between the camera and the tool end reference position, and the workpiece coordinate system source information. S2. Based on the work area template, extract the workpiece work reference features, visible edges of the tool end and fixed edges of the fixture in the industrial work image frame, and form the image feature state based on the visible edge length, number of endpoints, gray scale jump amplitude, reflective coverage area, occlusion and break area, edge width and repeating texture distribution. S3. Based on short-time attitude data, the expected movement range of image features between adjacent industrial operation image frames is defined. Within the expected movement range of image features, the positional changes, edge direction changes, and endpoint relationship changes of similar image features are verified to form a visual inertial continuity relationship between the operation reference features, the visible edge of the tool end, and the fixed edge of the fixture. S4. Based on the image feature state, visual inertial continuity, workpiece coordinate system source information, tool end visible edge stability, and process tolerance, screen candidate poses of the tool end relative to the workpiece work reference, and exclude candidate poses that do not meet the correction conditions, do not enter the visual inertial continuity, do not meet the process tolerance for visible edge stability, or have a jump relative to the previous candidate pose that entered the correction output. S5. Based on the selected candidate poses, the previous output pose correction results, and the process tolerance, the translation correction amount and angle correction amount are generated, and the image frame number, inertial data segment number, image feature status, workpiece coordinate system source information, and anomaly markers are output together with the translation correction amount and angle correction amount.

[0007] Furthermore, in S1: After the current workpiece enters the workstation, the image processing industrial control computer creates a work batch table. The work batch table contains the work object number, work reference type, end camera physical sequence number, inertial measurement unit physical sequence number, fixture number, work area template number, camera and tool end reference position relationship table number, and workpiece coordinate system source information. The physical serial number of the end camera, the physical serial number of the inertial measurement unit, the template number of the work area, the number of the table showing the relationship between the camera and the tool end reference position, and the source information of the workpiece coordinate system are all bound to the same work batch table.

[0008] Furthermore, in S1: The image processing industrial control computer reads the difference between the trigger time of the industrial operation image frame and the sampling time of the short-time attitude data; When the difference is less than five milliseconds, the industrial operation image frame and short-time attitude data are entered into S2; When the difference reaches five milliseconds but is less than ten milliseconds, the industrial operation image frame is marked as a time critical frame. The time critical frame enters S2 and is used only for visual inertial continuity verification in S3. When the difference reaches ten milliseconds, the industrial operation image frame is marked as a time-asynchronous frame, and the time-asynchronous frame will not enter the subsequent correction process.

[0009] Furthermore, in S2: The work area templates provide the workpiece work reference feature template area, the tool end visible edge template area, and the fixture fixed edge template area, respectively. The image processing industrial control computer reads the candidate boundaries within the corresponding template area and determines the target image features based on the distance between the candidate boundaries and the template centerline, the length of continuous boundaries, and the degree of matching of endpoint relationships. When two or more candidate boundaries cannot uniquely determine the features of the target image, they are retained as multiple candidate boundaries and enter the repeated verification judgment.

[0010] Furthermore, in S2: The image feature status uses six status names: available, affected by reflection, affected by occlusion, affected by blur, duplicate pending verification, and unavailable. Only one image feature state is registered for the same target image feature; When the same target image feature simultaneously meets multiple state formation conditions, it is registered in the following order: unavailable, repeated verification pending, affected by blur, affected by occlusion, affected by reflection, and available. If the previous order is met, the next order will not be registered.

[0011] Furthermore, in S3: The expected range of movement of image features is formed by the target image feature position in the previous effective industrial operation image frame, short-time attitude segments, camera intrinsic parameter calibration results, lens working distance, and maximum short-time vibration amplitude of the workstation. The image processing industrial control computer defines the expected occurrence area of ​​the target image features in the current industrial operation image frame based on short-time attitude segments, and expands to both sides of the expected occurrence area with the pixel width corresponding to the maximum short-time vibration amplitude of the workstation to obtain the expected movement range of the image features.

[0012] Furthermore, in S3: Similar image feature verification is performed within the expected movement range of the image features; Verification of workpiece operation datum features with image features of the same operation datum type; verification of the visible edge of the tool end and the tool shell boundary within the tool end template area; verification of the fixed edge of the fixture and the straight line boundary within the fixture positioning surface template area. Candidate image features that fall within the expected movement range of image features, have an edge direction difference of less than three degrees, maintain consistent endpoint relationships, and have a gray-level jump amplitude difference of less than twenty-five gray-level values ​​are considered to enter the visual inertial continuity relationship.

[0013] Furthermore, in S4: The formation method of candidate pose is as follows: under the same workpiece coordinate system source information, the workpiece working reference feature is used as the working position reference, and the visible edge of the tool end and the relationship table between the camera and the tool end reference position are used as the tool end position reference to form the translational deviation and angular deviation of the tool end relative to the workpiece working reference. Translational deviation refers to the lateral and longitudinal deviations of the tool end reference relative to the workpiece working reference within the corresponding working plane of the image. Angular deviation refers to the directional difference between the tool end reference direction and the workpiece working reference direction.

[0014] Furthermore, in S5: The previous output pose correction result is the pose correction result that was most recently completed and output within the same batch of work, and the work object number, work reference type, work area template number, work coordinate system source information, end camera physical sequence number and inertial measurement unit physical sequence number corresponding to the pose correction result are all consistent with the current candidate pose. When the translational change between the current candidate pose and the previous output pose correction result is less than one-third of the process translation tolerance, and the angle change is less than one-third of the process angle tolerance, the image processing control computer generates the translational correction amount and angle correction amount for the current frame. If either the translational or angular change reaches half of the corresponding process tolerance, a jump anomaly flag is written to the current frame, and no new translational or angular corrections are output.

[0015] Compared with existing technologies, this invention provides a visual-inertial-based method for industrial work posture correction, which has the following advantages: 1. By simultaneously acquiring industrial operation image frames and short-term attitude data when the industrial operation equipment reaches the pre-photograph position, and combining the operation area template, the camera and tool end reference position relationship table, and the workpiece coordinate system source information, image feature states are formed for the workpiece operation reference features, the visible edges of the tool end, and the fixed edges of the fixture. Then, the expected movement range of image features between adjacent image frames is limited using short-term attitude data. Within this range, the positional changes, edge direction changes, and endpoint relationship changes of similar image features are verified, thereby filtering out erroneous candidate poses caused by reflection, occlusion, blurring, and duplicate textures. This processing chain addresses the problem that degraded image features and inertial short-term data can easily cause abrupt changes in the correction amount when they are used together in pose correction in industrial operation sites. It ensures that the candidate poses entering the correction output simultaneously satisfy the image state, visual inertial continuity, workpiece coordinate source, and tool end stability constraints, reducing the situation where erroneous image features are directly used to form translation and angle correction amounts.

[0016] 2. Furthermore, by outputting the image frame number, inertial data segment number, image feature status, workpiece coordinate system source information, and anomaly markers along with translation and angle correction values, each pose correction result has a clear image source, inertial data source, coordinate source, and anomaly source. For situations affected by reflections, occlusion, blurring, repeated verification, time asynchrony, and candidate pose jumps, the correction process can perform blocking, verification, and maintenance according to the corresponding state, avoiding the mixing of correction results from different sources, coordinate systems, and image quality states. This improves the continuity, verifiability, and stability of pose correction results before industrial operations. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the process of an industrial operation posture correction method based on visual inertia according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example: Figure 1 A vision-inertial-based method for industrial work pose correction is presented, which involves the following steps: S1. When the industrial operation equipment reaches the pre-photograph position, simultaneously collect the industrial operation image frame of the end camera, the short-time attitude data of the inertial measurement unit, the operation area template, the relationship table between the camera and the tool end reference position, and the workpiece coordinate system source information. S2. Based on the work area template, extract the workpiece work reference features, visible edges of the tool end and fixed edges of the fixture in the industrial work image frame, and form the image feature state based on the visible edge length, number of endpoints, gray scale jump amplitude, reflective coverage area, occlusion and break area, edge width and repeating texture distribution. S3. Based on short-time attitude data, the expected movement range of image features between adjacent industrial operation image frames is defined. Within the expected movement range of image features, the positional changes, edge direction changes, and endpoint relationship changes of similar image features are verified to form a visual inertial continuity relationship between the operation reference features, the visible edge of the tool end, and the fixed edge of the fixture. S4. Based on the image feature state, visual inertial continuity, workpiece coordinate system source information, tool end visible edge stability, and process tolerance, screen candidate poses of the tool end relative to the workpiece work reference, and exclude candidate poses that do not meet the correction conditions, do not enter the visual inertial continuity, do not meet the process tolerance for visible edge stability, or have a jump relative to the previous candidate pose that entered the correction output. S5. Based on the selected candidate poses, the previous output pose correction results, and the process tolerance, the translation correction amount and angle correction amount are generated, and the image frame number, inertial data segment number, image feature status, workpiece coordinate system source information, and anomaly markers are output together with the translation correction amount and angle correction amount.

[0020] S1 is specifically implemented as follows: When the industrial equipment reaches the pre-photograph position, the pre-photograph position is determined by the workpiece observation distance, camera field of view coverage, and tool end safety distance recorded in the work process card. After the station is debugged, this information is written into the work batch table of the image processing industrial control computer. The work batch table is created by the image processing industrial control computer after the current workpiece enters the station. The table contains the work object number, work reference type, end-camera physical sequence number, inertial measurement unit physical sequence number, fixture number, work area template number, camera and tool end reference position relationship table number, and workpiece coordinate system source information. This is used to ensure the consistency of the source of image frames, inertial data, and calibration data within the same batch.

[0021] After the industrial equipment reaches the pre-capture position, the equipment control cabinet outputs a hardware trigger pulse. This pulse simultaneously enters the trigger terminal of the end-effector camera and the synchronization input terminal of the inertial measurement unit (IMU). The end-effector camera acquires an industrial operation image frame, and the IMU acquires short-term attitude data adjacent to that image frame. The end-effector camera is an industrial camera fixed near the work tool. The industrial operation image frame originates from the camera acquisition card and carries the frame number, trigger time, exposure time, and end-effector camera physical serial number.

[0022] The inertial measurement unit is fixed on the end-effector camera mounting bracket and is rigidly connected to the end-effector camera. The short-time attitude data comes from the angular velocity data and linear acceleration data within 100 milliseconds before and after the trigger time of the industrial operation image frame. The short-time attitude data carries the inertial data segment number, sampling time, and physical serial number of the inertial measurement unit.

[0023] Before the short-term attitude data enters the subsequent processing, the image processing industrial control computer reads the inertial zero bias meter and performs zero bias subtraction on the angular velocity data and linear acceleration data. The inertial zero bias meter is formed by sampling for sixty seconds while the working equipment is stationary and is bound to the physical serial number of the inertial measurement unit.

[0024] The image frame rate is set at 60 frames per second, and the inertial data sampling frequency is set at 400 Hz. These parameters are derived from the instruction manuals of the workstation camera, the inertial measurement unit (IMU) manual, and the equipment's no-load sampling results. The verification criteria are: during the pre-image positioning of 30 consecutive qualified test pieces, the tool end movement between adjacent industrial operation image frames can be covered by no fewer than six inertial sampling points, and the IMU does not experience any consecutive 20-millisecond outputs reaching 90% of its maximum range in the pre-image positioning position. If the industrial operation image frame rate and inertial data sampling frequency are lower than the above criteria, the image processing control computer stops generating new pose correction outputs and writes an insufficient acquisition frequency warning.

[0025] The work area template originates from the workstation calibration process. The template marks the pixel areas of the workpiece's work reference features, the visible edges of the tool end, and the fixture's fixed edges in the normal photographing position. The work area template is formed by combining the camera's intrinsic parameter calibration plate image, the fixture reference hole image, and the first qualified image of the workpiece.

[0026] The template area for the workpiece's operational reference features is determined based on the operational reference type. For welding stations, it corresponds to the weld edge and lap edge; for gluing stations, it corresponds to the gluing path edge and hole / groove edge; for grinding stations, it corresponds to the boundary to be processed and the workpiece's outer contour edge; and for assembly and positioning stations, it corresponds to the hole / groove edge and assembly reference line. The template area for the visible edge of the tool end is formed from the tool end reference calibration plate image, and the template area for the fixture fixing edge is formed from the fixture reference hole image and the fixture reference edge image. If the operational area template does not correspond to the current workpiece number, fixture number, and end-camera physical sequence number, the current industrial operation image frame will not enter S2.

[0027] The table showing the relationship between the camera and the tool's reference position is derived from the image of the tool's reference calibration plate. The tool's reference calibration plate is fixed to the end of the working tool, and it has three non-collinear circular holes. The center distance between these holes is given in the coordinate measuring machine (CMM) report. The image processing industrial control computer uses the positions of the three hole centers in the industrial work image and the spatial positions of the holes in the CMM report to create a table showing the relationship between the camera coordinate system and the tool's reference coordinate system.

[0028] The relationship table is bound to the end-point camera's physical serial number, tool end number, and calibration date. If any of the following occurs: end-point camera disassembly / assembly, lens replacement, camera mounting bracket adjustment, or failure of the tool end reference calibration board retest, the original relationship table will cease to be used for the current work batch.

[0029] The workpiece coordinate system source information is either the fixture reference hole image source or the fixed workpiece coordinate system source. The fixture reference hole image source is formed by two positioning holes and one reference edge within the current industrial operation image frame. When both positioning holes and one reference edge fall into the corresponding pixel area of ​​the operation area template, the image processing industrial control computer marks the workpiece coordinate system source information as the fixture reference hole image source.

[0030] The fixed workpiece coordinate system source is generated from the tooling acceptance report and is only activated when the center position difference of the positioning holes of twenty consecutively clamped workpieces in the same fixture is less than one-third of the allowable tolerance for translation in the operation process, and the difference in the direction of the reference edge is less than one-third of the allowable tolerance for angle in the operation process. If no reference hole image source for the fixture is generated and the fixed workpiece coordinate system source does not meet the above acceptance criteria, the current industrial operation image frame will not enter the pose correction output process.

[0031] The image processing industrial control computer reads the difference between the trigger time of the industrial operation image frame and the sampling time of the short-time attitude data. When the difference is less than five milliseconds, the industrial operation image frame and the short-time attitude data enter S2. When the difference reaches five milliseconds but is less than ten milliseconds, the image processing industrial control computer reads the adjacent inertial sampling points before and after the trigger time to form a time-critical short-time attitude segment, and marks the current industrial operation image frame as the time-critical frame; the time-critical frame enters S2, but in S3 it can only be used for visual inertial continuity verification and cannot form a new pose correction output independently.

[0032] When the difference reaches ten milliseconds, the current industrial operation image frame is marked as a time-asynchronous frame and will not proceed to the subsequent correction process. When three consecutive time-asynchronous frames appear, the image processing industrial control computer stops generating new pose correction outputs and writes clock check prompts to the camera trigger line, inertial synchronization line, and acquisition card.

[0033] The aforementioned short-term attitude data window from the first 100 milliseconds to the last 100 milliseconds, the 60-second static sampling, the 5-millisecond and 10-millisecond synchronization difference thresholds, and the processing method of using the time critical frame only for visual inertial continuity verification, are derived from the camera trigger delay, the inertial measurement unit synchronization input delay, the acquisition card clock jitter record, the inertial zero bias sampling record of the working equipment in the stopped state, and the synchronous verification results of 30 consecutive qualified test pieces.

[0034] S2 is specifically implemented as follows: Based on the work area template obtained in S1, the workpiece work reference features, visible edges of the tool end, and fixture fixing edges are read from the industrial work image frame. The industrial work image frame entering S2 is the same image frame that entered subsequent processing in S1. The image processing industrial control computer first retains the original industrial work image frame, and then converts the original industrial work image frame into an eight-bit grayscale image. Subsequent image feature readings are all completed on this eight-bit grayscale image. The original industrial work image frame is only used for retention and verification and is not directly changed.

[0035] The work area template provides the workpiece work reference feature template area, the tool end visible edge template area, and the fixture fixed edge template area. The image processing industrial control computer only reads the boundary within the corresponding template area and does not perform supplementary searches outside the template area.

[0036] The workpiece operation reference features are one or two of the following: workpiece outer contour edge, hole / groove edge, weld edge, and assembly reference line. The specific type is given by the operation reference type in the operation process card. The welding station reads the weld edge and lap edge, the glue application station reads the glue application path edge and hole / groove edge, the grinding station reads the boundary to be processed and the workpiece outer contour edge, and the assembly positioning station reads the hole / groove edge and assembly reference line.

[0037] The visible edge of the tool end originates from the boundary between the tool shell and the background within the tool end template area, while the fixed edge of the fixture originates from the straight boundary between the fixture positioning edge and the background within the fixture positioning surface template area. The distinction between the workpiece operation datum feature, the visible edge of the tool end, and the fixed edge of the fixture is based on the area location, boundary type, and adjacent reference relationship registered in the operation area template. Specifically, the adjacent reference for the visible edge of the tool end is the tool shell, the adjacent reference for the fixed edge of the fixture is the fixture positioning surface, and the adjacent reference for the workpiece operation datum feature is the workpiece body contour, holes, welds, or assembly surfaces. If the template area does not correspond to the current work object number, fixture number, or operation datum type, the current industrial operation image frame will not proceed to the subsequent state formation process.

[0038] The image processing industrial control computer first extracts candidate boundaries within each template area, and then determines the target image features. Candidate boundaries are boundary segments within the same template area that satisfy the grayscale jump condition and extend continuously. The template length is the registered length of the target boundary within that template area in the completed workstation calibration state, derived from the pixel length records of the corresponding boundaries in the first qualified workpiece image, the fixture reference hole image, and the tool end reference calibration plate image.

[0039] The target image features are determined from candidate boundaries in the following order: first, the distance between the candidate boundary and the template centerline is compared; then, the lengths of consecutive boundaries are compared; and finally, the degree of endpoint relationship matching is compared. The image processing control computer prioritizes selecting the candidate boundary with the smallest distance from the template centerline; when the distance between the candidate boundary and the template centerline are the same, the candidate boundary with the longest consecutive boundary length is selected; when both the distance between the candidate boundary and the template centerline and the length of consecutive boundaries are the same, the candidate boundary with the most endpoint relationship matches is selected; if the target image features still cannot be uniquely determined, one candidate boundary is not forcibly selected, but multiple candidate boundaries are retained and repeatedly checked during subsequent state formation.

[0040] The image processing industrial control computer reads the visible edge length, number of endpoints, grayscale jump amplitude, reflective coverage area, occlusion and break area, edge width, and repeating texture distribution of the target image within the template area. All read objects come from the same industrial operation image frame, without cross-frame supplementation. The visible edge length is the length of the continuous boundary that satisfies the grayscale jump condition.

[0041] The grayscale jump condition is that the average grayscale difference of the three-pixel-width area on both sides of the boundary reaches the eighteenth grayscale value. The eighteenth grayscale value comes from the tenth percentile value of the grayscale difference between the workpiece reference edge and the background area in thirty qualified test pieces, and is verified by the boundary of the black and white grid calibration board.

[0042] The number of endpoints is determined by the connection between the boundary end and the intersections of the hole / groove, weld end, assembly edge, and outer contour. A valid endpoint is defined as the boundary end falling within the endpoint tolerance range registered in the template. The endpoint tolerance range is a five-pixel radius region surrounding the center of the template endpoint; this parameter is derived from the maximum positional fluctuation of the same endpoint within ten consecutive frames in the workstation calibration image. The grayscale jump amplitude is the average grayscale difference between three-pixel-width regions on either side of the target image feature.

[0043] The reflective coverage area is the pixel area with a grayscale value of 240 that continuously covers the work reference area. The 240 value comes from the near-saturated imaging range of the eight-bit grayscale image and has been verified by the workstation lighting exposure curve. The occlusion break area is the area where the work reference feature is interrupted within the template range and the break boundary connects with the tool end contour, fixture contour, and workpiece external foreign object contour. The workpiece external foreign object contour is a newly added boundary that does not belong to the workpiece body contour, tool shell contour, or fixture positioning surface contour.

[0044] The edge width is the pixel width of the grayscale transition band that spans the boundary. Under normal focusing conditions, it is verified by a 0.1 mm blade calibration piece to be no more than six pixels. Combined with the vibration margin at the workstation, eight pixels are used as the blur discrimination threshold. The repetitive texture distribution is formed by the number of grayscale transition bands with similar directions and spacing within the same template area. When the directional difference of four consecutive grayscale transition bands is less than three degrees and the spacing difference is less than four pixels, it enters the repetitive texture distribution judgment.

[0045] The aforementioned three-dimensional difference and four-pixel spacing difference are derived from the verification results of repeated grinding texture, repeated weld images and periodic hole and groove edge images in thirty qualified test pieces. The verification standard is that this value can distinguish between the real operation reference boundary and the repeated texture boundary.

[0046] The image processing industrial control computer forms image feature states based on the read objects. These states are named using six categories: available, affected by reflection, affected by occlusion, affected by blur, pending verification (repeated), and unavailable. Only one image feature state is registered for the same target image feature. When the same target image feature simultaneously meets multiple state formation conditions, it is registered in the following order: unavailable, pending verification (repeated), affected by blur, affected by occlusion, affected by reflection, and available. Once a previous state is met, subsequent states are not registered.

[0047] The order was set based on the fact that the preceding state interfered more with the subsequent visual inertial continuous verification and pose selection than the following state, and this was verified by thirty qualified test pieces, five artificial occlusion test pieces, five artificial reflection test pieces, and five artificial defocus test pieces.

[0048] The usable state corresponds to a target image feature whose visible length reaches 70% of the template length, has at least two endpoints, exhibits 18 levels of grayscale abrupt changes, has an edge width between two and eight pixels, and whose reflected light coverage area is less than 12% of the target image feature's area, and does not contain multiple candidate boundaries. The reflected light-affected state corresponds to a target image feature whose reflected light coverage area overlaps with the target image feature, whose uncovered edge length reaches 40% of the template length, and which contains one endpoint.

[0049] The occlusion-affected state corresponds to the occlusion break area being connected to the tool end contour, fixture contour, and workpiece foreign object contour. The difference in direction between the two edges of the break area is less than three degrees, and the extension directions of the two boundary segments before and after the break still point to the same template centerline. The blur-affected state corresponds to the target image feature with an edge width exceeding eight pixels and a visible length reaching 60% of the template length.

[0050] The "Duplicate to be Verified" state corresponds to the situation where there are multiple candidate boundaries with similar directions within the same template area and the number of endpoints is less than two, or the target image features cannot be uniquely determined even according to the target image feature determination order. The "Unavailable" state corresponds to one of the following situations: the visible length is less than 40% of the template length, the number of endpoints is zero, the reflective coverage area reaches 35% of the feature area, or the edge width continuously exceeds eight pixels and the endpoints cannot be identified.

[0051] The template length (70%, 60%, and 40%) and the feature area (12% and 35%) are derived from the image verification results of 30 qualified test pieces, 5 artificially occluded test pieces, 5 artificially reflected test pieces, and 5 artificially defocused test pieces. The verification standard is that, using the above values, the first piece's pass rate corresponds to the stability of image feature reading, and it is able to distinguish between normal operation images, partially reflected images, partially occluded images, defocused and blurred images, and repetitive texture images without additional manual correction.

[0052] After the image processing industrial control computer forms the image feature state, it writes the target image feature number, corresponding template region number, image feature state, visible length, number of endpoints, grayscale jump amplitude, reflective coverage area ratio, occlusion and disconnection area ratio, edge width, and repeating texture mark into the image feature table of the current industrial operation image frame, and sends the image feature table to S3 for visual inertial continuity relationship verification.

[0053] S3 is implemented as follows: Based on the short-term attitude data obtained in S1, the expected movement range of image features between adjacent industrial operation image frames is defined. Within the expected movement range of image features, the positional changes, edge direction changes, and endpoint relationship changes of similar image features are verified to form a visual inertial continuity relationship between the operation reference features, the visible edge of the tool end, and the fixed edge of the fixture. The data entering S3 includes the image feature table of the current industrial operation image frame formed in S2, the image feature table of the previous valid industrial operation image frame, the short-term attitude data obtained in S1, the camera intrinsic parameter calibration results, the operation area template, and the table of the relationship between the camera and the tool end reference position.

[0054] A valid industrial operation image frame is an image frame that has completed the formation of S2 image feature states and has not been marked as a time-asynchronous frame, a completely black frame, a completely white frame, a corrupted image file frame, or a frame with missing inertial data. When the current industrial operation image frame is a time-critical frame, the current image frame can enter the visual-inertial continuity relationship verification, but its verification result is only used to judge feature continuity and does not form a new pose correction output independently.

[0055] The image processing industrial control computer first reads the trigger time interval between the current industrial operation image frame and the previous valid industrial operation image frame, and then reads the short-time attitude data within that time interval. The input quantities of the short-time attitude data are the angular velocity data and linear acceleration data within that time interval, and the input quantities are defined as the same inertial measurement unit physical serial number, the same inertial zero offset table, and the same operation batch.

[0056] The image processing industrial control computer subtracts the corresponding zero bias from the inertial zero bias table from the short-time attitude data to form short-time attitude segments. These short-time attitude segments are used to indicate the change in the end-camera's line-of-sight direction and the short-term movement trend of the tool end between two adjacent frames. If the duration of a short-time attitude segment is less than 80% of the trigger time interval between two adjacent frames, the current industrial operation image frame does not form a new visual-inertial continuity, and an inertial segment insufficiency marker is added.

[0057] When the angular velocity data reaches 90% of the inertial measurement unit's range for 20 consecutive milliseconds, the current industrial operation image frame does not form a new visual inertial continuity relationship, and an inertial impact marker is written. The aforementioned 20 milliseconds and 90% are derived from equipment no-load sampling and mounting bracket impact tests. The verification caliber is that normal pre-photograph position movement does not trigger this marker, while mounting bracket collisions and inertial measurement unit loosening can trigger this marker.

[0058] The expected movement range of the image features is jointly formed by the target image feature position in the previous valid industrial operation image frame, short-term attitude segments, camera intrinsic parameter calibration results, lens working distance, and the maximum short-term vibration amplitude of the workstation. The target image feature position in the previous valid industrial operation image frame comes from the image feature table written by S2, the short-term attitude segments come from the inertial data between the current two frames, the lens working distance comes from the workpiece observation distance recorded in the operation process card, and the maximum short-term vibration amplitude of the workstation comes from the pre-photograph position operation data of thirty consecutive qualified test pieces.

[0059] Based on the trend of gaze direction change given by the short-term posture segment, the image processing industrial control computer defines the expected appearance area of ​​the target image features within the corresponding work area template of the current industrial operation image frame, and expands to both sides of the expected appearance area with the pixel width corresponding to the maximum short-term vibration amplitude of the workstation to obtain the expected movement range of the image features.

[0060] The pixel width corresponding to the maximum short-term vibration amplitude of the workstation is calculated from the pixel length of the five-millimeter grid in the calibration board image. The conversion input is the camera intrinsic parameter calibration result and the lens working distance. The conversion calibrator is the same end camera, the same lens focal length, and the same pre-take-off position.

[0061] If the maximum short-term vibration amplitude of the workstation is not obtained in the current batch of work, the image processing industrial control computer reads the most recent workstation vibration record that passed the first piece review; if there is no workstation vibration record that passed the first piece review, the current industrial work image frame will not enter the pose correction output process.

[0062] Similar image feature verification is performed within the expected movement range of the image features, and continuous relationships are not established outside this range. Workpiece operation reference features are only verified against image features of the same operation reference type; visible edges of the tool end are only verified against the tool shell boundary within the tool end template area; and fixture fixed edges are only verified against the straight line boundary within the fixture positioning surface template area. The image processing industrial control computer reads candidate similar image features falling within the expected movement range of the image features from the image feature table of the current industrial operation image frame, and sequentially verifies changes in position, edge direction, endpoint relationships, and grayscale jump amplitude. Position changes are based on the current candidate similar image feature falling within the expected movement range of the image features.

[0063] Edge direction changes are defined as follows: the directional difference between the current candidate image features and the corresponding target image features in the previous valid industrial operation image frame is less than three degrees. Endpoint relationship changes are defined as follows: the connection type of the hole-groove intersection, weld end point, assembly datum line intersection, outer contour intersection, and fixture positioning edge end point remains consistent.

[0064] The grayscale jump amplitude change is defined as a grayscale jump amplitude difference between the current candidate image features and the corresponding target image features in the previous valid industrial operation image frame being less than 25 levels. The three-dimensional direction difference is derived from the upper limit of the continuous change of the working reference edge direction in 30 qualified test pieces, combined with the tool end vibration margin. The 25-level grayscale value is derived from the upper limit of the inter-frame fluctuation of the grayscale difference of the workpiece boundary under stable lighting conditions at the same workstation.

[0065] When only one candidate image feature in the current industrial operation image frame simultaneously satisfies the verifications for position change, edge direction change, endpoint relationship change, and grayscale jump amplitude change, this candidate image feature and its corresponding target image feature in the previous valid industrial operation image frame enter into a visual inertial continuity relationship. If two or more candidate image features in the current industrial operation image frame simultaneously satisfy position change and edge direction change, but only one candidate image feature maintains the same endpoint relationship, the candidate image feature with the consistent endpoint relationship enters into a visual inertial continuity relationship.

[0066] When two or more candidate similar image features in the current industrial operation image frame simultaneously satisfy position change, edge direction change, and endpoint relationship change, the image processing industrial control computer compares the grayscale jump difference between each candidate similar image feature and the corresponding target image feature in the previous valid industrial operation image frame. The candidate similar image feature with the smallest grayscale jump difference and below the 25th grayscale level is entered into the visual inertial continuity relationship. When two or more candidate similar image features have the same grayscale jump difference and cannot be uniquely matched, the visual inertial continuity relationship is not established, and a duplicate pending verification continuity mark is written.

[0067] In the current industrial operation image frame, when candidate similar image features fall within the expected movement range of image features, but the endpoint relationship is missing and the edge direction difference is less than five degrees, the candidate similar image features enter the verification continuity relationship. The verification continuity relationship is not directly used as the basis for candidate pose selection. Only when both the current and subsequent clear frames point to the same boundary will it be transferred to the visual inertial continuity relationship.

[0068] The aforementioned 80% short-term attitude segment integrity threshold and five-degree orientation difference threshold are derived from the adjacent frame inertial sampling coverage records of thirty consecutive qualified specimens, the boundary orientation verification results of artificially occluded specimens, and the operation data of the pre-photograph position.

[0069] Image feature status is used for visual inertial continuity verification. When the image feature status is available, the target image feature can directly participate in continuity verification. When the image feature status is affected by reflection, the image processing control computer only uses the edge segments not covered by reflection and the still visible endpoints for continuity verification; the boundaries within the reflection-covered area do not participate in continuity verification. When the image feature status is affected by occlusion, the image processing control computer reads the edge segments on both sides of the occluded and broken area. Only when both edge segments fall within the expected movement range of the image feature and the difference in direction between the two edges is less than three degrees, is the image feature allowed to enter the continuity relationship to be verified.

[0070] When an image feature is in a state of being blurred, it will not be included in the visual inertial continuity relationship. When an image feature is in a state of being repeatedly pending verification, it will only be allowed to enter the visual inertial continuity relationship if candidate boundaries with the same endpoint relationship exist within the same template region in both the previous and subsequent valid industrial operation image frames. When an image feature is in a state of being unavailable, it will not participate in the continuity verification.

[0071] The stability of the visible edge of the tool end is synchronously established in S3. The image processing industrial control computer reads the visible edge of the tool end that enters the visual inertial continuity relationship in the current industrial operation image frame and verifies the position and edge direction changes of the visible edge of the tool end with those in the previous valid industrial operation image frame. When the visible edge of the tool end is within the expected movement range of the image features, the edge direction change is less than three degrees, and the offset between the visible edge of the tool end and the center line of the tool end template is less than half of the allowable translation tolerance of the operation process, the stability of the visible edge of the tool end is marked as meeting the correction conditions.

[0072] If the visible edge of the tool end does not enter the visual inertial continuity relationship, or if the offset between the visible edge of the tool end and the center line of the tool end template reaches half of the allowable translation tolerance of the operation process, the stability of the visible edge of the tool end is marked as not meeting the correction conditions. If the visible edge of the tool end is not visible for five consecutive frames, the image processing industrial control computer writes a tool end visible edge missing marker, which is used in S4 to restrict candidate poses from entering the correction output.

[0073] The continuity of the fixture fixed edge is synchronously established in S3. When the workpiece coordinate system source information is the fixture reference hole image source, the fixture fixed edge must enter the visual inertial continuity relationship; if the fixture fixed edge does not enter the visual inertial continuity relationship, the current industrial operation image frame is not considered a valid frame for the fixture reference hole image source. When the workpiece coordinate system source information is a fixed workpiece coordinate system source, the continuity of the fixture fixed edge is written into the current image frame as verification information, but is not a necessary condition for the current frame to enter S4. The fixed workpiece coordinate system source is only enabled after the tooling acceptance criteria are met in S1, and this enabling condition is not relaxed again in S3.

[0074] After the image processing industrial control computer completes continuous verification, it writes the continuous markers of the work reference features, the stability of the visible edges at the tool end, the continuous markers of the fixture fixed edges, the continuous relationships to be verified, the repeated continuous markers to be verified, the markers of insufficient inertial segments, and the markers of inertial impact into the visual inertial continuity table of the current industrial work image frame. The visual inertial continuity table is bound to the image frame number, the inertial data segment number, the work area template number, the work object number, and the work batch number, and is sent to S4 for candidate pose screening.

[0075] S4 is implemented as follows: Based on the image feature state formed by S2, the visual inertial continuity relationship formed by S3, the workpiece coordinate system source information obtained by S1, the stability of the visible edge of the tool end, and the process tolerance, candidate poses of the tool end relative to the workpiece work reference are selected. The inputs for the candidate poses are the workpiece work reference features that enter the visual inertial continuity relationship, the visible edge of the tool end, the fixture fixed edge, the camera and tool end reference position relationship table, the workpiece coordinate system source information, and the process tolerance.

[0076] The candidate pose is formed by using the workpiece work reference features as the work position reference under the same workpiece coordinate system source information, and using the visible edge of the tool end and the relationship table between the camera and the tool end reference position as the tool end position reference, to form the translational deviation and angular deviation of the tool end relative to the workpiece work reference.

[0077] Translational deviation refers to the lateral and longitudinal deviations of the tool end reference relative to the workpiece working reference within the corresponding working plane of the image. Angular deviation refers to the directional difference between the tool end reference direction and the workpiece working reference direction. Translational and angular deviations do not read the joint angles of the working equipment or motion control commands; they are derived solely from industrial work image frames, the camera-tool end reference position relationship table, and workpiece coordinate system information.

[0078] When the workpiece coordinate system source information is the fixture reference hole image source, the image processing industrial control computer reads two positioning holes and one reference edge from the current industrial operation image frame. If both positioning holes and the reference edge are within visual inertial continuity, the current image frame meets the conditions for being a fixture reference hole image source. If one of the two positioning holes is not within visual inertial continuity, or if the reference edge is not within visual inertial continuity, the current image frame is not considered a valid frame for being a fixture reference hole image source.

[0079] When the workpiece coordinate system source information is a fixed workpiece coordinate system source, the image processing industrial control computer reads the clamping repeatability data from the tooling acceptance report, and the image feature status of the workpiece operation datum features is required to be available. The fixed workpiece coordinate system source is only activated when the difference in the center position of the positioning holes of twenty consecutively clamped workpieces in the same fixture is less than one-third of the allowable tolerance for translational operation and the difference in the direction of the datum edge is less than one-third of the allowable tolerance for angular operation. These twenty workpieces are derived from the continuous clamping records during the tooling acceptance stage, and the verification standard is that both the first and last piece inspections meet the corresponding operation process tolerances.

[0080] The allowable tolerances for the work process are derived from the process card, work instructions, tooling acceptance report, and first-piece inspection report of trial production. At the welding station, the allowable values ​​for weld center deviation and welding torch angle deviation are read; at the adhesive application station, the allowable values ​​for nozzle center deviation and nozzle posture deviation are read; at the grinding station, the allowable values ​​for contact center deviation and normal angle deviation are read; and at the assembly and positioning station, the allowable values ​​for hole center deviation and insertion angle deviation are read.

[0081] When the process tolerance does not correspond to the current work object number, work reference type, and fixture number, the candidate pose will not be included in the correction output. The stability of the visible edge of the tool end is given by the stability of the visible edge of the tool end in S3. When the offset between the visible edge of the tool end and the center line of the tool end template reaches half of the process translation tolerance, or when the visible edge of the tool end does not enter the visual inertial continuity relationship, the stability of the visible edge of the tool end does not meet the correction conditions.

[0082] Candidate pose selection is performed sequentially based on image feature state, visual inertial continuity, workpiece coordinate system source information, tool edge visible stability, and candidate pose continuity. If any of the preceding conditions is not met, the current candidate pose will not proceed to the next selection. If the image feature state is unavailable, the corresponding candidate pose will not be included in the correction output. If the image feature state is affected by blur, the corresponding candidate pose will not be included in the correction output.

[0083] When the image feature state is affected by reflection, if the length of the edge not covered by reflection reaches 40% of the template length and there is a valid endpoint, the corresponding candidate pose will proceed to subsequent screening; if the length of the edge not covered by reflection is less than 40% of the template length, or the number of valid endpoints is zero, the corresponding candidate pose will not be included in the correction output. When the image feature state is affected by occlusion, if both edges of the occluded break area are included in the continuity to be verified, the directional difference between the two edges of the occluded break area is less than three degrees, and the number of consecutive occlusion frames does not exceed two frames, the corresponding candidate pose will proceed to subsequent screening; if the number of consecutive occlusion frames reaches three frames, the corresponding candidate pose will not be included in the correction output.

[0084] When the image feature state is "repeated and awaiting verification," the current candidate pose first enters the "pose awaiting verification" state and does not enter the correction output. When both the previous and subsequent valid industrial operation image frames have candidate boundaries with the same endpoint relationship within the same template region, and these candidate boundaries both enter the visual inertial continuity relationship, the pose awaiting verification state proceeds to subsequent filtering. Before the subsequent valid industrial operation image frame is formed, the pose awaiting verification state is retained, and no new pose correction results are output.

[0085] During the visual-inertial continuity screening, candidate poses whose work reference features are not included in the visual-inertial continuity are not included in the correction output. If the visible edge of the tool end is not included in the visual-inertial continuity, and the image feature state of the workpiece work reference features is not available, the corresponding candidate pose is not included in the correction output. If the fixture fixed edge is not included in the visual-inertial continuity, and the workpiece coordinate system source information is from the fixture reference hole image, the corresponding candidate pose is not included in the correction output.

[0086] When the workpiece coordinate system source information is a fixed workpiece coordinate system source, the continuity of the fixture fixed edge is written into the current image frame as verification information. It is not a necessary condition for the candidate pose to enter the subsequent screening, but the workpiece operation reference feature must be in an available state and enter the visual inertial continuity relationship.

[0087] During candidate pose continuity screening, the image processing industrial control computer reads the candidate pose that last entered the calibration output. The candidate pose that last entered the calibration output is the most recent candidate pose that entered the calibration output within the same job batch, and the job object number, job reference type, workpiece coordinate system source information, and job area template number corresponding to the candidate pose are all consistent with the current candidate pose.

[0088] If no candidate pose from the previous calibration output exists in the current batch of work, the current candidate pose is used as the initial candidate pose. It will only be included in the calibration output if the workpiece reference features corresponding to the current candidate pose are in an available state, the reference features are in a visual inertial continuity relationship, the visible edge stability of the tool end meets the calibration conditions, and the workpiece coordinate system source information is valid. The initial candidate pose does not use jump filtering.

[0089] The rule for a jump in candidate pose relative to the previous candidate pose entering the correction output is that the change between the translation deviation of the current candidate pose and the translation deviation of the previous candidate pose entering the correction output reaches half of the process translation tolerance, or the change between the angle deviation of the current candidate pose and the angle deviation of the previous candidate pose entering the correction output reaches half of the process angle tolerance.

[0090] Translational and angular changes are both generated under the same workpiece coordinate system and are not compared across workpiece coordinate systems. The aforementioned half-threshold is derived from the pre-operation review results of thirty qualified test pieces and five manually set deviation test pieces. The verification criterion is that candidate poses exceeding this threshold cannot stably correspond to the first piece's qualified inspection state before re-shooting.

[0091] Candidate poses that undergo a jump are not included in the correction output and are written with a jump mark. After the next valid industrial operation image frame is formed, the image feature state filtering, visual inertial continuity filtering, workpiece coordinate system source information filtering, tool end visible edge stability filtering, and candidate pose continuity filtering are re-executed.

[0092] After the image processing industrial control computer completes the screening, it writes the candidate poses that will enter the correction output, the pose status to be verified, the reasons for not entering the correction output, the workpiece coordinate system source information, the stability of the visible edges at the tool end, the process tolerance number, and the jump mark into the candidate pose screening table. The candidate pose screening table is bound to the image frame number, inertial data segment number, work object number, work reference type, work area template number, and work batch number, and is sent to S5 to generate translation correction and angle correction amounts.

[0093] S5 is implemented as follows: Based on the candidate poses selected in S4 and entering the correction output, the previous outputtable pose correction result, and the process tolerance, translational and angular correction values ​​are generated. The image frame number, inertial data segment number, image feature status, workpiece coordinate system source information, and anomaly markers are output along with the translational and angular correction values. The data entering S5 comes from the candidate pose selection table generated in S4. Candidate poses not entering the correction output in the candidate pose selection table do not generate translational and angular correction values; they are only written into the work pose correction result table as the pose status to be verified, the reason for not entering the correction output, and anomaly markers.

[0094] The previous output pose correction result is the most recently completed pose correction result within the same batch of work, and the corresponding work object number, work reference type, work area template number, workpiece coordinate system source information, end-camera physical sequence number, and inertial measurement unit physical sequence number are all consistent with the current candidate pose. If there are inconsistencies among these objects, the previous output pose correction result is not read, and the current candidate pose is processed according to the initial output aperture.

[0095] When there is no previous outputtable pose correction result in the current batch of work, the candidate pose that enters the correction output is used as the initial candidate pose. The image feature status corresponding to the initial candidate pose is available. The work reference feature enters the visual inertial continuity relationship. The stability of the visible edge at the tool end meets the correction conditions. When the source information of the workpiece coordinate system is valid, the initial translation correction amount and the initial angle correction amount are formed.

[0096] The input for translation correction is the lateral and longitudinal deviations in the candidate poses entering the correction output, the translation correction in the previous output pose correction result, and the translation tolerance in the process tolerance. The input quantity is the planar deviation of the tool end relative to the workpiece work reference in the same workpiece coordinate system. The joint angles of the workpiece are not read, and the motion path of the workpiece is not read.

[0097] The input for angle correction is the directional deviation in the candidate pose entering the correction output, the angle correction in the previous output pose correction result, and the angle tolerance in the process tolerance. The input quantity is the angle difference between the tool end direction and the workpiece work reference direction in the same workpiece coordinate system, and does not read the motion control commands of the work equipment. Both translation correction and angle correction are based on the workpiece coordinate system source information and are not compared across the fixture reference hole image source and the fixed workpiece coordinate system source.

[0098] When the current frame generates the correction amount, the lateral correction amount and the longitudinal correction amount are taken as the inverse vectors of the lateral deviation and longitudinal deviation in the current candidate pose, respectively, and the angle correction amount is taken as the inverse vector of the direction deviation in the current candidate pose. When holding the output for a short time, the translation correction amount and angle correction amount in the previous output pose correction result are used.

[0099] When the translational change between the current candidate pose and the previous outputtable pose correction result is less than one-third of the translation tolerance, and the angular change is less than one-third of the angular tolerance, the image processing control computer generates the translational and angular correction values ​​for the current frame. If the translational change reaches one-third but is less than half of the translation tolerance, and the angular change is less than half of the angular tolerance, the current frame proceeds to the next frame for verification, and no new correction values ​​are output.

[0100] If either the translational or angular change reaches half of the corresponding process tolerance, a jump anomaly flag is written to the current frame, and no new correction amount is output. After the next valid industrial operation image frame is formed, the image processing control computer rereads the candidate pose selection table of S4 and re-executes the output judgment of S5.

[0101] One-third threshold is used for the stable output range, and half threshold is used for the abnormal blocking range. Both thresholds are derived from trial run verification data of thirty qualified test pieces and five manually deviated test pieces from the same workstation. The verification criteria are: the thirty qualified test pieces, under this threshold, can continuously output translational and angular correction amounts corresponding to the first piece's qualified state; and the five manually deviated test pieces can trigger a pending verification or jump anomaly marker when the deviation approaches the process tolerance boundary. After any of the following occurs: adjustment of the process tolerance, fixture replacement, end-effector camera disassembly / assembly, inertial measurement unit replacement, or tool end reference recalibration, the one-third and half thresholds are reconfirmed through trial run verification.

[0102] When an image's feature state is affected by reflection and passes the S4 screening, the image processing industrial control computer generates translational and angular correction values ​​and writes them to a reflection anomaly marker. The reflection anomaly marker is saved along with the length of the edge not covered by reflection, the number of effective endpoints, and the proportion of the reflected coverage area. When an image's feature state is affected by occlusion and passes the S4 screening, if the number of consecutive occlusion frames does not exceed two frames, the image processing industrial control computer uses the previous output pose correction result to generate a short-term hold output and writes it to an occlusion hold anomaly marker; if the number of consecutive occlusion frames reaches three frames, it stops outputting new translational and angular correction values ​​and writes an occlusion exceedance anomaly marker.

[0103] Short-term hold output only uses the translation and angle correction values ​​from the previous output pose correction result, and does not use the currently affected image features to regenerate correction values. The boundaries for short-term hold output are: stopping output after three consecutive frames, the visible edge at the tool end becoming invisible for five consecutive frames, the number of consecutive occlusion frames not exceeding two frames, and the number of consecutive occlusion frames reaching three frames to form an occlusion over-limit anomaly mark. These boundaries are derived from the operation data of the same workstation's pre-position photography, the results of manual occlusion test piece verification, and the work cycle requirements. The verification standard is that short-term occlusion does not trigger error correction, and continuous occlusion can block the output of new correction values.

[0104] When an image feature is in one of the following states: affected by blur, pending verification, or unavailable, no new correction value is output for the current frame, and the corresponding state is written to an anomaly flag. If the image is pending verification and cannot transition to a visual-inertial continuity relationship after the next valid industrial operation image frame is completed, a "failed to repeat verification" anomaly flag is written.

[0105] Image frame numbers come from the camera acquisition card, inertial data segment numbers come from the inertial measurement unit time stamp, image feature status comes from the image feature table formed in S2, workpiece coordinate system source information comes from the job batch table formed in S1, and candidate pose selection results come from the candidate pose selection table formed in S4.

[0106] Anomalies are marked from time-critical frames, time-asynchronous frames, insufficient inertial segments, inertial shock, affected by reflection, occlusion retention, occlusion exceeding limits, affected by blur, repeated verification pending, repeated verification failed, unusable, unstable visible edge of tool end, continuous failure of fixture fixed edge, and abrupt state.

[0107] The image frames corresponding to the following conditions will not output new translation and angle correction amounts: out-of-time frames, insufficient inertial segments, inertial shock, blurred image, repeated verification failure, unusable image, unstable visible edges at the tool end, continuous failures of the fixture fixed edge, and jump state.

[0108] The output data is written to the work pose correction result table of the image processing industrial control computer. The work pose correction result table saves the work batch number, work object number, work reference type, image frame number, inertial data segment number, end camera physical sequence number, inertial measurement unit physical sequence number, work area template number, workpiece coordinate system source information, image feature status, translation correction amount, angle correction amount, anomaly mark, and output time.

[0109] The translational and angular correction values ​​in the job pose correction result table are only used as pre-job calibration inputs; the output does not include robot joint angles, robot trajectory points, or motion control commands. After the industrial work equipment reads the translational correction values, angular correction values, and anomaly markers, the equipment control system completes the pre-execution calibration according to the existing process flow.

[0110] In a welding station engineering verification, the work object is the lap weld of plate parts. The allowable value for weld center deviation recorded on the work process card is 0.3 mm, the allowable value for welding torch angle deviation is 1.5 degrees, the end-point camera frame rate is 60 frames per second, and the inertial measurement unit sampling frequency is 400 Hz. After the workpiece enters the fixture, the image processing industrial control computer establishes a work batch table, writing in the work object number, weld edge work reference type, end-point camera physical sequence number, inertial measurement unit physical sequence number, fixture number, work area template number, and the number of the camera and tool end reference position relationship table.

[0111] After the industrial operation equipment reaches the pre-photograph position, the equipment control cabinet outputs a hardware trigger pulse, the end camera collects industrial operation image frames of the weld area, and the inertial measurement unit synchronously collects short-term attitude data from 100 milliseconds before to 100 milliseconds after the image frame trigger time.

[0112] The image processing industrial control computer reads the weld edge and lap edge within the weld edge template area, the welding gun shell edge within the tool end template area, and the fixture fixing edge within the fixture positioning surface template area. In the current image frame, if the visible length of the weld edge reaches 82% of the template length, there are two endpoints, the grayscale jump reaches 23 levels, the edge width is four pixels, and the reflective coverage area is less than 12% of the area where the weld edge is located, the image processing industrial control computer registers the weld edge as usable.

[0113] In the next adjacent image frame, the weld edge is locally affected by metal reflection. The area covered by the reflection overlaps with the weld edge. The length of the edge not covered by the reflection reaches 51% of the template length, and there is still a valid endpoint. The image processing industrial control computer registers the weld edge as being affected by reflection.

[0114] The image processing industrial control computer reads short-time attitude data between two adjacent frames, subtracts the inertial zero bias to form a short-time attitude segment, and limits the expected movement range of the weld edge image features based on the short-time attitude segment, camera intrinsic parameter calibration results, workpiece observation distance, and maximum short-time vibration amplitude of the workstation. Weld edge segments in image frames affected by reflection that are not covered by reflection are included in the expected movement range of image features. If the edge direction change is less than three degrees and the endpoint relationship remains consistent with the previous valid image frame, the image processing industrial control computer includes this weld edge in the visual inertial continuity relationship.

[0115] The edge of the welding torch shell is located within the visible edge template area of ​​the tool end, and its offset relative to the center line of the tool end template is less than half of the allowable value of the weld center deviation. The stability of the visible edge of the tool end meets the correction conditions.

[0116] The image processing industrial control computer forms candidate poses based on the weld edge, visible edge of the tool end, fixed edge of the fixture, the reference position relationship table between the camera and the tool end, and the workpiece coordinate system source information, which are based on the continuous relationship of the visual inertial. The translational change of the candidate pose relative to the candidate pose output by the previous correction is less than one-third of the allowable value of the weld center deviation, and the angular change is less than one-third of the allowable value of the welding torch angular deviation, and no jump in candidate pose occurs.

[0117] The image processing industrial control computer inputs the candidate pose into the correction output, forming translation correction and angle correction amounts. It also writes the image frame number, inertial data segment number, reflection status, fixture reference hole image source, reflection anomaly mark, translation correction and angle correction amounts into the work pose correction result table.

[0118] In another image frame at the same welding station, the welding torch nozzle obscures a local area of ​​the weld edge. The difference in direction between the two edges of the obscured area is less than three degrees. The obscuration occurs in two consecutive frames. The image processing control computer registers the weld edge as being affected by the obscuration and generates a short-term hold output when both the preceding and following valid image frames point to the same weld edge.

[0119] When maintaining output for a short period of time, the image processing industrial control computer uses the lateral correction amount, longitudinal correction amount, and angle correction amount from the previous output pose correction result, and does not regenerate new correction amounts based on the weld edge under the current occlusion state.

[0120] When the number of consecutive occlusion frames reaches three, the image processing industrial control computer stops outputting new translation and angle correction values ​​and writes an occlusion exceeding limit anomaly flag. During the engineering verification process, the translation and angle correction values ​​are all derived from industrial operation image frames, short-term posture data, operation area templates, camera and tool end reference position relationship tables, and workpiece coordinate system source information; the joint angles of the operating equipment, robot trajectory points, and motion control commands are not read.

[0121] The above content is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for correcting pose during industrial operations based on visual inertia, characterized in that, Perform the following steps: S1. When the industrial operation equipment reaches the pre-photograph position, simultaneously collect the industrial operation image frame of the end camera, the short-time attitude data of the inertial measurement unit, the operation area template, the relationship table between the camera and the tool end reference position, and the workpiece coordinate system source information. S2. Based on the work area template, extract the workpiece work reference features, visible edges of the tool end and fixed edges of the fixture in the industrial work image frame, and form the image feature state based on the visible edge length, number of endpoints, gray scale jump amplitude, reflective coverage area, occlusion and break area, edge width and repeating texture distribution. S3. Based on short-time attitude data, the expected movement range of image features between adjacent industrial operation image frames is defined. Within the expected movement range of image features, the positional changes, edge direction changes, and endpoint relationship changes of similar image features are verified to form a visual inertial continuity relationship between the operation reference features, the visible edge of the tool end, and the fixed edge of the fixture. S4. Based on the image feature state, visual inertial continuity, workpiece coordinate system source information, tool end visible edge stability, and process tolerance, screen candidate poses of the tool end relative to the workpiece work reference, and exclude candidate poses that do not meet the correction conditions, do not enter the visual inertial continuity, do not meet the process tolerance for visible edge stability, or have a jump relative to the previous candidate pose that entered the correction output. S5. Based on the selected candidate poses, the previous output pose correction results, and the process tolerance, the translation correction amount and angle correction amount are generated, and the image frame number, inertial data segment number, image feature status, workpiece coordinate system source information, and anomaly markers are output together with the translation correction amount and angle correction amount.

2. The method for industrial work posture correction based on visual inertia according to claim 1, characterized in that, In S1: After the current workpiece enters the workstation, the image processing industrial control computer creates a work batch table. The work batch table contains the work object number, work reference type, end camera physical sequence number, inertial measurement unit physical sequence number, fixture number, work area template number, camera and tool end reference position relationship table number, and workpiece coordinate system source information. The physical serial number of the end camera, the physical serial number of the inertial measurement unit, the template number of the work area, the number of the table showing the relationship between the camera and the tool end reference position, and the source information of the workpiece coordinate system are all bound to the same work batch table.

3. The method for industrial work posture correction based on visual inertia according to claim 1, characterized in that, In S1: The image processing industrial control computer reads the difference between the trigger time of the industrial operation image frame and the sampling time of the short-time attitude data; When the difference is less than five milliseconds, the industrial operation image frame and short-time attitude data are entered into S2; When the difference reaches five milliseconds but is less than ten milliseconds, the industrial operation image frame is marked as a time critical frame. The time critical frame enters S2 and is used only for visual inertial continuity verification in S3. When the difference reaches ten milliseconds, the industrial operation image frame is marked as a time-asynchronous frame, and the time-asynchronous frame will not enter the subsequent correction process.

4. The method for industrial work posture correction based on visual inertia according to claim 1, characterized in that, In S2: The work area templates provide the workpiece work reference feature template area, the tool end visible edge template area, and the fixture fixed edge template area, respectively. The image processing industrial control computer reads the candidate boundaries within the corresponding template area and determines the target image features based on the distance between the candidate boundaries and the template centerline, the length of continuous boundaries, and the degree of matching of endpoint relationships. When two or more candidate boundaries cannot uniquely determine the features of the target image, they are retained as multiple candidate boundaries and enter the repeated verification judgment.

5. The method for industrial work posture correction based on visual inertia according to claim 4, characterized in that, In S2: The image feature status uses six status names: available, affected by reflection, affected by occlusion, affected by blur, duplicate pending verification, and unavailable. Only one image feature state is registered for the same target image feature; When the same target image feature simultaneously meets multiple state formation conditions, it is registered in the following order: unavailable, repeated verification pending, affected by blur, affected by occlusion, affected by reflection, and available. If the previous order is met, the next order will not be registered.

6. The method for industrial work posture correction based on visual inertia according to claim 1, characterized in that, In S3: The expected range of movement of image features is formed by the target image feature position in the previous effective industrial operation image frame, short-time attitude segments, camera intrinsic parameter calibration results, lens working distance, and maximum short-time vibration amplitude of the workstation. The image processing industrial control computer defines the expected occurrence area of ​​the target image features in the current industrial operation image frame based on short-time attitude segments, and expands to both sides of the expected occurrence area with the pixel width corresponding to the maximum short-time vibration amplitude of the workstation to obtain the expected movement range of the image features.

7. The method for industrial work posture correction based on visual inertia according to claim 1, characterized in that, In S3: Similar image feature verification is performed within the expected movement range of the image features; Verification of workpiece operation datum features with image features of the same operation datum type; verification of the visible edge of the tool end and the tool shell boundary within the tool end template area; verification of the fixed edge of the fixture and the straight line boundary within the fixture positioning surface template area. Candidate image features that fall within the expected movement range of image features, have an edge direction difference of less than three degrees, maintain consistent endpoint relationships, and have a gray-level jump amplitude difference of less than twenty-five gray-level values ​​are considered to enter the visual inertial continuity relationship.

8. The method for industrial work posture correction based on visual inertia according to claim 1, characterized in that, In S4: The formation method of candidate pose is as follows: under the same workpiece coordinate system source information, the workpiece working reference feature is used as the working position reference, and the visible edge of the tool end and the relationship table between the camera and the tool end reference position are used as the tool end position reference to form the translational deviation and angular deviation of the tool end relative to the workpiece working reference. Translational deviation refers to the lateral and longitudinal deviations of the tool end reference relative to the workpiece working reference within the corresponding working plane of the image. Angular deviation refers to the directional difference between the tool end reference direction and the workpiece working reference direction.

9. The method for industrial work posture correction based on visual inertia according to claim 8, characterized in that, In S5: The previous output pose correction result is the pose correction result that was most recently completed and output within the same batch of work, and the work object number, work reference type, work area template number, work coordinate system source information, end camera physical sequence number and inertial measurement unit physical sequence number corresponding to the pose correction result are all consistent with the current candidate pose. When the translational change between the current candidate pose and the previous output pose correction result is less than one-third of the process translation tolerance, and the angle change is less than one-third of the process angle tolerance, the image processing control computer generates the translational correction amount and angle correction amount for the current frame. If either the translational or angular change reaches half of the corresponding process tolerance, a jump anomaly flag is written to the current frame, and no new translational or angular corrections are output.

Images