Image Extraction-Based Package Damage Detection Method
By employing multi-source image fusion and path search techniques, the accuracy problem of detecting package damage under complex packaging surfaces was solved, enabling continuous extraction and scale measurement of damage boundaries.
Patent Information
- Application Number
- CN202611256881.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-19
- Publication Date
- 2026-11-17
AI Technical Summary
Existing package damage detection technologies struggle to accurately extract true damage boundaries under complex packaging surface conditions, especially in cases of texture interference, discontinuous features, and complex tears, leading to misjudgments and boundary mismatches.
By acquiring visible light images, depth images, and ray transmission images, spatial registration is performed, linear textures are extracted, and surface normal variations and transmission attenuation variations are combined to screen out discontinuous candidate lines. Minimum path search and skeleton connection are then performed, boundary points are corrected, and package damage detection results are generated.
It can effectively distinguish between pseudo-linear features and real material damage, restore the continuity of the damage path, reduce misjudgment, improve the completeness and quantifiability of damage detection, and reflect the location, area and scale of damage.
Smart Images

Figure CN123118490A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image detection technology, and in particular to a method for detecting package damage based on image extraction. Background Technology
[0002] With the development of automated logistics sorting and machine vision inspection technology, package appearance quality inspection is gradually shifting from manual observation to image-based and digital inspection methods. Current package damage detection typically uses visible light imaging to acquire color, texture, and edge information of the packaging surface, and utilizes grayscale changes, edge features, and texture features to locate cracks, tears, and surface anomalies. For packaging surfaces with obvious dents or creases, depth imaging can be combined to obtain the surface spatial morphology. X-ray transmission imaging can reflect the differences in transmission attenuation after rays pass through the packaging material, used to observe changes in the packaging material and the internal structure of the package. With the development of multi-source imaging technology, visible light images, depth images, and X-ray transmission images are increasingly being applied to multi-dimensional image analysis of the same inspection object.
[0003] Existing technologies still have certain limitations under complex packaging surface conditions. Printed lines, tape edges, reflective areas, and color boundaries on the packaging surface can easily form linear features similar to material cracks in visible light images. Relying solely on two-dimensional texture or edge information makes it difficult to determine whether these linear features correspond to actual material structural changes. Furthermore, actual cracks, when affected by local occlusion, reflection, and weakened image response, tend to appear as discontinuous line segments. Conventional connected domain or local edge tracking struggles to maintain the continuity of the actual damage path. For curved or bifurcated damage, directly searching the edges along a fixed direction may mispair the boundaries of different damage locations, thus affecting the closure of the damaged area and the determination of dimensional information such as damage width and boundary length. Summary of the Invention
[0004] In view of the aforementioned existing problems, the present invention is proposed.
[0005] Therefore, this invention provides an image extraction-based method for package damage detection to solve the problem of difficulty in continuously and accurately extracting the true damage boundary of a package under conditions of complex texture interference and discontinuous damage.
[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a package damage detection method based on image extraction, comprising: The visible light image, depth image, and ray transmission image of the package to be detected are acquired, spatial registration is performed, and the main body region of the package is extracted to obtain the registered package image group and pixel mapping relationship; Linear textures are extracted from the registered image group, and the linear textures are filtered by combining surface normal variation and transmission attenuation variation to obtain a set of discontinuous candidate lines with the main extension direction. The path search area is determined based on the main extension direction of the discontinuous candidate line set. The minimum path search is performed by combining the surface normal change and the local extreme value of transmission attenuation, and the candidate line segments are connected to obtain a continuous broken candidate skeleton with a local propagation direction. Image cut-outs are established along the local propagation direction of the continuous broken candidate skeleton. Skeleton segments are screened by combining surface structure changes and transmission structure continuity to obtain the material cracked and broken skeleton and surface deformation area. An orthogonal boundary search line is established along the cracked and damaged skeleton of the material, and the paired boundary points are corrected according to the boundary extension direction. The damaged area and damage scale are determined by combining the surface deformation area and pixel mapping relationship, and the package damage detection results are generated.
[0007] As a preferred embodiment of the image extraction-based package damage detection method of the present invention, the acquisition of the registration of package image groups and pixel mapping relationships specifically includes: The visible light image and depth image are transformed according to the calibration relationship of the imaging device to obtain a spatially aligned visible light image and depth image; Based on the spatial positional relationship between the packaged surface and the bearing reference surface in the spatially aligned depth image, the packaged main body region is separated, and the depth pixels within the packaged main body region are converted into surface spatial positions; The surface spatial position is projected onto the X-ray transmission image, and the corresponding image area is cropped according to the main body area of the package to obtain the registered package image group; Pixel correspondences between visible light images, depth images, and ray transmission images are established based on surface spatial location and projection location, forming pixel mapping relationships.
[0008] As a preferred embodiment of the image extraction-based package damage detection method of the present invention, the step of extracting linear textures from the registered package image group specifically includes: The visible light images in the registered image group are converted into grayscale images. Gabor filtering is then applied to the grayscale images along the horizontal, vertical, and two diagonal directions to obtain multi-directional texture responses. By comparing the multi-directional texture responses at the same pixel location, retaining the direction with the most obvious response, determining the pixel texture direction, and performing non-maximum suppression along the direction perpendicular to the pixel texture direction, the texture response with width is refined into a single pixel line, resulting in a linear texture.
[0009] As a preferred embodiment of the image extraction-based package damage detection method of the present invention, the acquisition of the discontinuous candidate line set specifically includes: The corresponding positions of the linear texture in the depth image and the ray transmission image are determined according to the pixel mapping relationship. The surface normal changes of the neighborhood on both sides of the linear texture are calculated, and the transmission attenuation changes along the normal direction of the linear texture are extracted. Remove linear textures where the surface normal and transmission attenuation remain continuous, and retain line segments where the surface normal and transmission attenuation change to obtain candidate line segments. The endpoints and intersections are determined according to the pixel connectivity of the candidate line segments, and the main extension direction is determined along the continuous direction of the candidate line segments, resulting in a set of discontinuous candidate lines with the main extension direction.
[0010] As a preferred embodiment of the image extraction-based package damage detection method of the present invention, the step of determining the path search region based on the main extension direction of the discontinuous candidate line set specifically includes: Select the endpoints of candidate line segments from the set of discontinuous candidate lines, and determine the search direction in front of the endpoints according to the main extension direction of the corresponding candidate line segments; Expand the search range of the candidate line segment endpoints along the search direction, retaining adjacent pixels that are continuous with the main extension trend of the candidate line segment, to obtain the path search area.
[0011] As a preferred embodiment of the image extraction-based package damage detection method of the present invention, the acquisition of the continuous damage candidate skeleton specifically includes: The surface normal abrupt change and transmission attenuation local extrema at the corresponding positions of each path node in the path search area are detected according to the pixel mapping relationship. Path nodes with two structural changes are determined as the first passable node, path nodes with only one structural change are determined as the second passable node, and path nodes that only maintain visible light direction response are determined as the third passable node. The path propagation order is determined according to the order of the first passable node, the second passable node, and the third passable node. The path is extended step by step from the endpoint of the current candidate line segment to the endpoint of the adjacent candidate line segment according to the path propagation order, and the path is restricted to propagate along the main extension trend of the current candidate line segment, so as to obtain the reachable path connecting the current candidate line segment and the adjacent candidate line segment. When there are multiple reachable paths between two candidate line segments, determine the number of turns and the propagation direction when entering the adjacent candidate line segment for each reachable path. Retain the reachable path with the fewest turns and the propagation direction that is consistent with the main extension direction of the adjacent candidate line segment to obtain the endpoint connection path. When there are intersections in the set of discontinuous candidate lines, the endpoint connection paths are extended from the intersections to the endpoints of the corresponding candidate line segments, the common paths that are repeatedly passed are merged, and the local propagation direction is determined to obtain a continuous broken candidate skeleton with a local propagation direction.
[0012] As a preferred embodiment of the image extraction-based package damage detection method of the present invention, the acquisition of the material cracking and damage skeleton and the surface deformation area specifically includes: The local propagation direction of the skeleton points is determined sequentially along the continuously damaged candidate skeleton. Image cut-outs perpendicular to the corresponding local propagation direction are established through each skeleton point to obtain the skeleton cut-out sequence. Based on the pixel mapping relationship, determine the linear texture change, surface normal change and transmission attenuation change at the corresponding position of the skeleton cut-off sequence to obtain the skeleton cut-off structure features. Retain the skeleton segments that simultaneously have continuous linear texture discontinuity, transmission attenuation discontinuity and surface normal change to obtain the material cracking and damage skeleton. Based on the structural characteristics of the skeleton cross-section, the area where the continuous transmission structure and the depth surface form a continuous folding deformation along the continuous skeleton points to the same side is defined as the surface deformation area.
[0013] As a preferred embodiment of the image extraction-based package damage detection method of the present invention, the establishment of orthogonal boundary search lines specifically includes: The skeleton segments without intersections are divided according to the pixel connectivity of the material cracking and damage skeleton, and the local propagation direction is determined according to the positional relationship between the current skeleton point and the adjacent skeleton points in each skeleton segment. With each skeleton point as the center, extend the search lines to both sides of the skeleton in a direction perpendicular to the corresponding local propagation direction to obtain orthogonal boundary search lines.
[0014] As a preferred embodiment of the image extraction-based package damage detection method of the present invention, the step of correcting the boundary points according to the boundary extension direction specifically includes: Search for the nearest edge position in the visible light image from the corresponding skeleton point to both sides along the orthogonal boundary search line to obtain paired boundary points; Select paired boundary points corresponding to adjacent skeleton points according to the skeleton propagation order, connect adjacent boundary points on the same side respectively, and determine the extension direction of the two sides of the boundary. Compare the extension directions of the two sides of the boundary with the local propagation direction of the corresponding skeleton point. For boundary points with inconsistent directions, move them to the front and back sides of the corresponding boundary towards the skeleton propagation direction, and retain the boundary positions where the boundary extension direction is consistent with the local propagation direction. For curved skeletons, the orthogonal boundary search line is updated according to the local propagation direction of continuous skeleton points, and the paired boundary points are continuously corrected according to the boundary extension direction. For bifurcated skeletons, the corresponding paired boundary points are corrected along the local propagation direction of each branch skeleton. When the visible light boundary is interrupted on one side, the candidate boundary position is determined along the surface normal abrupt change line at the corresponding position. The candidate boundary position is then verified using the continuous transmission attenuation change in the X-ray transmission image. The verified candidate boundary position is then used to fill in the corresponding side boundary point to obtain the corrected paired boundary point.
[0015] As a preferred embodiment of the image extraction-based package damage detection method of the present invention, the generation of package damage detection results specifically includes: By sequentially connecting the corrected paired boundary points and closing the boundary according to the local propagation direction of the material cracking and damage skeleton, the material cracking and damage area is obtained. The region is closed by continuously turning the boundary along the surface deformation region, thus obtaining the surface deformation and damage region. The damaged area of the package is determined by combining the cracked and damaged areas of the material and the deformed and damaged areas of the surface, and the damaged area and boundary of the package are transformed to the actual spatial location through pixel mapping relationship; The local damage width is determined based on the converted paired boundary points, and the maximum damage width is extracted. The damage boundary length is determined along the converted damage boundary, and the maximum surface deformation depth is determined in combination with the surface deformation damage area to generate the package damage detection result.
[0016] The beneficial effects of this invention are as follows: By utilizing the spatial correspondence between visible light texture, surface depth structure, and X-ray transmission structure, it can distinguish between printed lines, tape edges, and pseudo-linear features formed by changes in lighting and actual material damage, reducing misjudgments caused by surface texture interference in single visible light detection; for cases where damage is affected by reflection, occlusion, or weakened local response, resulting in discontinuous features, it can restore continuous damage paths based on the damage extension trend and reduce erroneous connections between adjacent non-homogeneous line segments, thereby maintaining the overall continuity of complex cracks; for damage involving bending, bifurcation, and local boundary interruptions, it can constrain the correspondence between the two sides of the boundary using the local propagation direction and the boundary extension direction, reducing the impact of boundary mismatch on damage width and area closure results. Simultaneously, this invention can distinguish between material cracking and surface deformation, and transform the damage boundary in the image to its actual spatial location, enabling the detection results to not only reflect the damage location and area but also obtain the maximum damage width, damage boundary length, and maximum surface deformation depth, improving the completeness and quantifiability of package damage detection results. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart of a package damage detection method based on image extraction.
[0019] Figure 2 A schematic diagram for multi-source image registration and construction of discontinuous candidate lines.
[0020] Figure 3 A schematic diagram illustrating the construction of candidate skeletons for continuous damage and the classification of skeleton segments.
[0021] Figure 4 A schematic diagram for searching and correcting the damaged boundary and quantifying the damaged scale. Detailed Implementation
[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.
[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a package damage detection method based on image extraction, including the following steps: S1. Acquire the visible light image, depth image, and ray transmission image of the package to be detected, perform spatial registration, and extract the main body region of the package to obtain the registered package image group and pixel mapping relationship.
[0026] S1.1. While the package to be inspected remains in the same inspection position, the visible light imaging device acquires the color and texture information of the surface of the package to be inspected to form a visible light image; the depth imaging device acquires the depth information corresponding to each pixel position on the surface of the package to be inspected to form a depth image; and the ray imaging device acquires the transmission attenuation distribution formed after the rays pass through the package to be inspected to form a ray transmission image.
[0027] Geometric calibration was performed on the visible light imaging device, the depth imaging device, and the ray imaging device to obtain the imaging intrinsic parameters of each imaging device. The rotation and translation relationships of the visible light imaging device relative to the depth imaging device and the ray imaging device relative to the depth imaging device were also calibrated. The imaging intrinsic parameters, rotation relationships, and translation relationships of each imaging device constitute the imaging device calibration relationship. Among them, the imaging intrinsic parameters are used to determine the correspondence between imaging coordinates and image pixel positions, and the rotation and translation relationships are used to determine the coordinate transformation relationship of the same spatial position between different imaging device coordinate systems.
[0028] Based on the rotation and translation relationships between the visible light imaging device and the depth imaging device, the spatial position corresponding to the depth image is transformed to the coordinate system of the visible light imaging device, and then projected onto the visible light image according to the imaging intrinsic parameters of the visible light imaging device. This establishes a positional correspondence between the visible light image pixels and the depth image pixels corresponding to the same spatial position, resulting in a spatially aligned visible light image and depth image.
[0029] It should be noted that by first aligning the visible light image with the depth image, the subsequent linear texture position and the surface depth position can maintain the same spatial orientation, avoiding mismatch between texture position and surface position due to differences in imaging perspective when processing the visible light image and the depth image separately.
[0030] S1.2. Based on the spatial positional relationship between the wrapping surface and the bearing reference surface in the spatially aligned depth image, separate the wrapping main body region and convert the depth pixels in the wrapping main body region into surface spatial positions.
[0031] The reference surface is the surface beneath the package being inspected when it is placed. The depth imaging device simultaneously acquires depth information from both the package surface and the reference surface. Based on the spatial position corresponding to the reference surface, it distinguishes between the image region on the reference surface and the image region on the package surface above the reference surface from the spatially aligned depth image, and deletes depth pixels belonging to the reference surface.
[0032] After deleting the depth pixels of the bearing reference plane, the remaining effective depth pixels are marked as connected components according to the eight-neighbor relationship. Effective depth pixels with adjacent pixel positions and corresponding spatial continuous positions are grouped into the same connected component. The number of effective depth pixels contained in each connected component is compared. The connected component with the largest number of effective depth pixels is retained as the main body connected component. The outermost continuous pixels of the main body connected component are used to determine the scope of the package surface. The pixel area inside the scope of the package surface is determined as the main body area of the package.
[0033] Among them, the eight-neighbor relationship refers to the pixel adjacency relationship between the current depth pixel and eight pixels adjacent in the horizontal, vertical and two diagonal directions. Spatial position continuity means that adjacent effective depth pixels are all located above the bearing reference plane and remain connected in the depth image.
[0034] According to the imaging intrinsic parameters of the depth imaging device, the pixel coordinates and corresponding depth values of each depth pixel in the area of the main body are transformed to the coordinate system of the depth imaging device according to the pinhole imaging back projection relationship, so as to obtain the three-dimensional surface coordinates corresponding to each depth pixel. The surface spatial position corresponding to the area of the main body is determined by all the three-dimensional surface coordinates.
[0035] S1.3. Based on the positional relationship between the X-ray imaging device and the depth imaging device, the spatial position of the surface is transformed from the coordinate system of the depth imaging device to the coordinate system of the X-ray imaging device. Based on the geometric projection relationship of the X-ray imaging device, the spatial position of each surface is projected onto the corresponding pixel position in the X-ray transmission image, and the projection range of the wrapped surface position in the X-ray transmission image is determined.
[0036] The visible light image and depth image are cropped separately according to the main area of the package. The ray transmission image is then cropped according to the projection range of the package surface position in the ray transmission image. Image content outside the main area of the package in the visible light image, depth image and ray transmission image is deleted to obtain the package visible light image, package depth image and package transmission image. The package visible light image, package depth image and package transmission image are then used to form a registered package image group.
[0037] It should be noted that the surface spatial position is first transformed by coordinates and then subjected to ray geometric projection, so that the cropping range of the ray transmission image comes from the same package surface spatial position, rather than finding the package boundary again according to the grayscale changes of the ray transmission image. This ensures that the visible light image, depth image and ray transmission image correspond to the same package area.
[0038] S1.4. Based on the pixel coordinates retained when the surface spatial position is formed by back-projection of depth pixels, each surface spatial position is matched one-to-one with the depth pixel that generated the corresponding surface spatial position, and the depth image pixel position corresponding to the surface spatial position is determined.
[0039] For each surface spatial location, the surface spatial location is transformed from the coordinate system of the depth imaging device to the coordinate system of the visible light imaging device according to the rotation and translation relationship between the visible light imaging device and the depth imaging device. Then, the transformed spatial location is projected onto the visible light image according to the imaging intrinsic parameters of the visible light imaging device. The resulting pixel coordinates are the visible light image pixel positions corresponding to the same surface spatial location.
[0040] For the same surface spatial location, according to the rotation and translation relationship between the X-ray imaging device and the depth imaging device, the surface spatial location is transformed to the coordinate system of the X-ray imaging device, and then projected onto the X-ray transmission image according to the geometric projection relationship of the X-ray imaging device. The resulting pixel coordinates are the pixel positions of the X-ray transmission image corresponding to the same surface spatial location.
[0041] Based on the same surface spatial location, a point-to-point correspondence is established between the determined visible light image pixel location, depth image pixel location, and ray transmission image pixel location, so that the same packaged surface spatial location corresponds to visible light texture information, surface depth information, and ray transmission attenuation information, forming a pixel mapping relationship; for surface spatial locations located outside the effective image range after projection, no corresponding pixel relationship is established; for surface spatial locations where effective pixel locations exist in the visible light image, depth image, and ray transmission image, the correspondence is retained.
[0042] S2. Extract linear textures from the registered image group, and filter linear textures by combining surface normal variation and transmission attenuation variation to obtain a set of discontinuous candidate lines with main extension direction.
[0043] S2.1. Convert the visible light images in the registered image group into grayscale images, and perform Gabor filtering on the grayscale images along the horizontal, vertical, and two diagonal directions to obtain multi-directional texture responses, specifically: The visible light images in the registered image group are converted pixel by pixel into single-channel grayscale images while keeping the image pixel coordinates unchanged, so that the pixel mapping relationship of the grayscale images remains consistent.
[0044] Four two-dimensional Gabor filters were established for the grayscale image, corresponding to horizontal, 45° diagonal, vertical, and 135° diagonal linear textures, respectively. The two-dimensional Gabor filters used a 21×21 pixel filter window, a wavelength of 5.66 pixels (corresponding to a spatial frequency of 0.177 cycles per pixel), a Gaussian standard deviation of 3.17 pixels, a spatial aspect ratio of 2, and a phase of 0. Since the direction parameter of the two-dimensional Gabor filter represents the normal direction of the filter fringes, the filter normal directions were set to 90°, 135°, 0°, and 45° for the horizontal, 45° diagonal, vertical, and 135° diagonal linear textures, respectively.
[0045] Two-dimensional Gaussian filtering uses Gaussian envelope modulation of the cosine function, expressed as: ; in: ; ; In the formula, Indicates the normal direction as The two-dimensional Gabor filter value; and This represents the pixel coordinates within the filtering window relative to the center of the window; and Represents the rotated pixel coordinates, in pixels; Indicates the normal direction of the two-dimensional Gabor filter; Indicates wavelength; Indicates the Gaussian standard deviation; Indicates the aspect ratio of a space.
[0046] A 21×21 pixel two-dimensional Gabor filter is sequentially applied to each pixel position in the grayscale image. The grayscale values within the filtering window are convolved with the two-dimensional Gabor filter values, and the absolute value of the convolution result is taken as the texture response in the corresponding direction. ; In the formula, Represents the pixel position in a grayscale image In the normal direction Texture response under; This represents the grayscale value at the corresponding position in a grayscale image. and Represents the pixel coordinates of a grayscale image.
[0047] After performing convolution in four directions, each grayscale image pixel corresponds to a texture response in four directions, forming a multi-directional texture response.
[0048] It should be noted that the four-directional two-dimensional Gabor filtering maintains the horizontal, vertical and two diagonal directions without changing the spatial coordinates of the registered package image group. Subsequently, the depth information and ray transmission attenuation information of the same position can be obtained directly using the formed pixel mapping relationship. Compared with extracting edges only according to gray-scale abrupt changes, the multi-directional texture response retains the directional information of the linear texture at the same time, providing a directional basis for distinguishing continuous linear structures on the packaging surface.
[0049] S2.2. Compare the multi-directional texture responses at the same pixel location to determine the pixel texture direction, and perform non-maximum suppression along the direction perpendicular to the pixel texture direction to obtain a linear texture, specifically: For each pixel position in the multi-directional texture response, compare the texture responses corresponding to horizontal, 45° diagonal, vertical, and 135° diagonal linear textures respectively, retain the texture response with the largest value, and determine the extension direction of the linear texture corresponding to the texture response with the largest value as the pixel texture direction.
[0050] The comparison direction for non-maximum suppression (NMS) is determined according to the pixel texture direction, and is perpendicular to the pixel texture direction. When the pixel texture direction is horizontal, the current pixel is compared with its two vertically adjacent pixels; when the pixel texture direction is vertical, the current pixel is compared with its two horizontally adjacent pixels; when the pixel texture direction is 45° diagonal, the current pixel is compared with its two 135° diagonal adjacent pixels; when the pixel texture direction is 135° diagonal, the current pixel is compared with its two 45° diagonal adjacent pixels.
[0051] If the maximum texture response of the current pixel is not less than the maximum texture responses of its two adjacent pixels on either side of the comparison direction, and is greater than the maximum texture response of at least one of its adjacent pixels, the current pixel is retained; otherwise, the texture response of the current pixel is set to zero. Pixels located at the boundary of the grayscale image are compared only with adjacent pixels within the effective image range. After traversing the grayscale image in the same way, the directional texture response with a certain width is compressed into a single-pixel line extending along a local maximum location, resulting in a linear texture.
[0052] It should be noted that non-maximum suppression directly uses the pixel texture direction to determine the comparison direction. Therefore, the thinning direction of the linear texture is consistent with the direction response obtained by Gabor filtering. There is no need to re-estimate the edge direction, and the obtained single-pixel line can continue to be used as a spatial positioning object for surface normal change and transmission attenuation change.
[0053] S2.3. Determine the corresponding position of the linear texture in the depth image and ray transmission image according to the pixel mapping relationship, calculate the surface normal change of the neighborhood on both sides of the linear texture, and extract the transmission attenuation change along the normal direction of the linear texture, specifically: According to the pixel mapping relationship, each linear texture pixel is mapped to a depth pixel position in the depth image; using the obtained surface spatial position, adjacent 3D surface coordinates are selected in the horizontal and vertical directions of the depth pixel position, and the local surface normal is calculated by the difference between the 3D coordinates in the two spatial directions. ; In the formula, Indicates the depth pixel position The corresponding unit surface normal; This represents the three-dimensional surface coordinates of the corresponding depth pixel.
[0054] The normal direction of the linear texture is determined according to the pixel texture direction. Adjacent depth pixels are selected on both sides of the linear texture pixel, and the unit surface normal on both sides is calculated. The angle between the unit surface normals on both sides is used as the change in surface normal.
[0055] In the formula, Indicates the change in surface normal on both sides of a linear texture; and These represent the unit surface normals on both sides of the linear texture.
[0056] Moving one pixel to each side along the normal direction of the linear texture, the change in surface normal at adjacent positions is obtained in the same way. If the change in surface normal at the linear texture position is greater than the change in surface normal at adjacent positions on both sides of the normal direction, the linear texture position is recorded as having a changed surface normal. If the change in surface normal at the linear texture position does not satisfy the aforementioned local maximum relation, the linear texture position is recorded as having a continuous surface normal. When the edge position of the depth image lacks the effective 3D surface coordinates required to calculate the unit surface normal, the surface normal on the missing side is not reconstructed; the corresponding linear texture position is then determined based on the ray transmission attenuation change.
[0057] Based on the pixel mapping relationship, the surface spatial positions of the same linear texture pixel and the two sides of the linear texture normal are respectively mapped to the ray transmission image to obtain the ray transmission attenuation value on both sides of the linear texture normal, and the absolute value of the transmission attenuation difference on both sides is calculated: ; In the formula, This indicates the amount of transmission attenuation change along the normal direction at the location of the linear texture; and 1 represents the ray transmission attenuation value corresponding to the adjacent positions on both sides of the linear texture normal; 2 corresponds to the interval of 2 pixels between the sampling positions on both sides. Therefore, the change in transmission attenuation represents the change in transmission attenuation corresponding to the unit pixel distance.
[0058] Continue moving one pixel along the normal direction of the linear texture and calculate the change in transmission attenuation at adjacent positions. If the change in transmission attenuation at the linear texture position is greater than the change in transmission attenuation at adjacent positions on both sides of the normal direction, the linear texture position is considered to have a change in transmission attenuation; if the change in transmission attenuation at the linear texture position does not satisfy the aforementioned local maximum relationship, the linear texture position is considered to have continuous transmission attenuation.
[0059] It should be noted that the surface normal variation and transmission attenuation variation are both calculated along the already determined linear texture normal direction and are limited to the same enclosing surface spatial position through pixel mapping relationship. Therefore, there is no process of finding the corresponding region again between the visible light texture, depth surface variation and ray transmission variation.
[0060] S2.4 Pixel-by-pixel inspection of surface normal variation and transmission attenuation variation results. When the surface normal and transmission attenuation remain continuous at the linear texture location, the corresponding linear texture pixel is deleted; when there is a surface normal variation at the linear texture location, the corresponding linear texture pixel is retained; when there is a transmission attenuation variation at the linear texture location, the corresponding linear texture pixel is retained. The retained linear texture pixels are connected according to the eight-neighbor pixel connectivity relationship, and the continuous pixels separated by the deleted pixels form candidate line segments.
[0061] Endpoints and intersections are determined based on the number of connected pixels in the eight-neighborhood of each candidate line segment. If a candidate line segment pixel has one pixel belonging to the same candidate line segment within its eight-neighborhood, the corresponding candidate line segment pixel is designated as an endpoint; if a candidate line segment pixel has at least three pixels belonging to the same candidate line segment within its eight-neighborhood, the corresponding candidate line segment pixel is designated as an intersection; if two pixels belonging to the same candidate line segment are within its eight-neighborhood, the corresponding candidate line segment pixel remains an internal pixel of the line segment.
[0062] For each candidate line segment, according to the preserved pixel texture direction, the maximum texture response in the horizontal, 45° diagonal, vertical, and 135° diagonal directions within the candidate line segment is accumulated, and the direction with the largest accumulated texture response is selected as the main extension direction of the candidate line segment. When the accumulated texture responses in two directions are the same, the two ends of the candidate line segment are connected, and the angles between the connecting lines and the two candidate directions are compared. The direction with the smaller angle is determined as the main extension direction.
[0063] The corresponding surface normal variation and transmission attenuation variation are preserved according to the pixel position of the candidate line segment, and the endpoints, intersections and main extension directions of the candidate line segment are recorded to obtain a set of discontinuous candidate lines with main extension directions.
[0064] It should be noted that, compared with directly retaining candidate lines based solely on edge intensity in visible light images, the direction of linear textures is first determined, and then linear textures that are continuous in both depth surfaces and transmission structures are deleted using surface normal changes and transmission attenuation changes at the same spatial location. This ensures that printed textures, illumination changes, and linear patterns on flat surfaces will not enter subsequent path searches based solely on visible light responses.
[0065] S3. Determine the path search area based on the main extension direction of the discontinuous candidate line set, perform minimum path search by combining surface normal abrupt change and local extreme value of transmission attenuation, and connect candidate line segments to obtain a continuous damaged candidate skeleton with local propagation direction.
[0066] S3.1. Sequentially select the endpoints of candidate line segments from the discontinuous candidate line set. Within the eight-neighborhood of each candidate line segment endpoint, determine the adjacent pixels belonging to the same candidate line segment. Use the direction from the adjacent pixels to the endpoint of the candidate line segment as the search direction for the candidate line segment to extend outward from the endpoint. The main extension direction of the candidate line segment is one of the horizontal, 45° diagonal, vertical, or 135° diagonal directions. The search direction is used to determine the side extending outward from the endpoint of the candidate line segment between two opposite extension directions corresponding to the main extension direction.
[0067] Starting from the endpoints of the candidate line segment, pixels are expanded layer by layer according to eight neighborhoods. Each expansion retains adjacent pixels directly in front of the search direction and in the two adjacent 45° directions, while deleting adjacent pixels opposite to the search direction or those laterally deviating from the main extension trend. For pixels obtained from subsequent expansions, the same directional relationship is continued to expand to the next layer, ensuring that the pixels participating in the search are always within the main extension trend range of the candidate line segment endpoints.
[0068] During the layer-by-layer expansion process, it is checked whether the newly added pixel belongs to the endpoints of other candidate line segments in the discontinuous candidate line set. The first other candidate line segment endpoint reached is determined as the endpoint of the adjacent candidate line segment, and further expansion beyond the expansion layer containing the endpoint of the adjacent candidate line segment stops. When multiple other candidate line segment endpoints appear in the same expansion layer, the search range extending from the current candidate line segment endpoint to each of the adjacent candidate line segment endpoints is retained. If the expansion reaches the effective image boundary of the registered image group but still does not reach other candidate line segment endpoints, the expansion of the current candidate line segment endpoint is terminated. All adjacent pixels retained between the current candidate line segment endpoint and the endpoints of the adjacent candidate line segments are determined as the path search region.
[0069] It should be noted that the search direction is directly derived from the endpoint positions and main extension direction of the candidate line segments. The search range is established layer by layer according to the original extension trend of the candidate line segments. It is not necessary to search for connection paths in the entire visible light image, thus limiting the image range processed by the subsequent minimum path search.
[0070] S3.2. Detect the surface normal abrupt change and local extremum of transmission attenuation at the corresponding positions of each path node in the path search area according to the pixel mapping relationship, divide the first passable node, the second passable node and the third passable node, and determine the path propagation order.
[0071] Here, path nodes represent the image pixel locations within the path search area that participate in candidate path expansion. Based on the pixel mapping relationship, the path nodes are mapped to pixel locations in the wrapping depth image and the wrapping transmission image, respectively.
[0072] For each path node, following the method for calculating surface normal changes, the unit surface normal is calculated using the 3D surface coordinates on both sides of the path node, and the change in surface normal on both sides of the path node is obtained. Then, the node is moved by 1 pixel along the corresponding direction, and the change in surface normal at the adjacent positions on both sides is obtained. When the change in surface normal corresponding to the path node is greater than the change in surface normal at the adjacent positions on both sides of the normal direction, the corresponding path node is determined to have a surface normal abrupt change; if the local maximum relation is not satisfied, the corresponding path node is determined to have no surface normal abrupt change.
[0073] Following the method for calculating transmission attenuation changes, the transmission attenuation values at corresponding positions on both sides of the path node in the wrapped transmission image are obtained, and the transmission attenuation change is calculated. The transmission attenuation changes at adjacent positions on both sides are then compared along the normal direction. When the transmission attenuation change of a path node is simultaneously greater than the transmission attenuation changes at adjacent positions on both sides of the normal direction, the corresponding path node is identified as having a local extremum of transmission attenuation; if the local maximum relationship is not satisfied, the corresponding path node is identified as not having a local extremum of transmission attenuation.
[0074] Path nodes that simultaneously exhibit both surface normal abrupt changes and local extrema of transmission attenuation are designated as first passable nodes; path nodes that only exhibit surface normal abrupt changes or only exhibit local extrema of transmission attenuation are designated as second passable nodes; for path nodes that do not exhibit both surface normal abrupt changes and local extrema of transmission attenuation, multi-directional texture responses are compared, and when the linear texture extension direction corresponding to the maximum texture response belongs to the allowed propagation direction, the corresponding path node is designated as the third passable node, and pixels that do not satisfy the allowed propagation direction do not participate in path expansion.
[0075] The path nodes that can be further expanded are arranged in the order of the first pass node, the second pass node, and the third pass node, so that the positions with both depth surface structure changes and ray transmission structure changes participate in path propagation first, the positions with a single structure change participate in path propagation next, and the positions that only maintain visible light direction response participate in path propagation last.
[0076] S3.3. Expand the path from the current candidate line segment endpoint to the adjacent candidate line segment endpoints in the path propagation order, perform minimum path search, and obtain the endpoint connection path.
[0077] A candidate path is established from the endpoint of the current candidate line segment. Following the defined path propagation order, all unvisited path nodes within the eight-neighborhood of the candidate path's endpoint are checked, and the path is extended forward according to the defined propagation direction. For each path node passed, the type of the access node and the propagation direction between adjacent path nodes are recorded, until the candidate path reaches the endpoint of the adjacent candidate line segment.
[0078] For multiple incomplete candidate paths, first compare the number of third-passage nodes in each candidate path; the candidate path with the fewest third-passage nodes continues to expand first. If the number of third-passage nodes is the same, then compare the number of second-passage nodes; the candidate path with fewer second-passage nodes continues to expand first. If the number of second-passage nodes is still the same, compare the total number of path nodes in each candidate path; the candidate path with the fewest total path nodes continues to expand first. Continue expanding in the aforementioned order, recording reachable paths that reach the endpoints of adjacent candidate line segments and have the same path propagation order.
[0079] For multiple reachable paths connecting the endpoints of the same current candidate line segment and the endpoints of adjacent candidate line segments, compare them segment by segment according to the propagation direction between adjacent path nodes. When the propagation direction of the current segment changes from the propagation direction of the next segment, record a turn and accumulate the number of turns for the corresponding reachable path. At the same time, determine the connection direction of the last two path nodes before the reachable path enters the endpoint of the adjacent candidate line segment, and compare the connection direction with the main extension direction of the adjacent candidate line segment.
[0080] Among the reachable paths whose propagation direction is consistent with the main extension direction of adjacent candidate line segments, the reachable path with the fewest turning times is retained to obtain the endpoint connection path; when multiple reachable paths have the same number of turning times, the reachable path with the fewest total number of path nodes is retained. If there is no reachable path whose propagation direction is consistent with the main extension direction of adjacent candidate line segments, no endpoint connection path is established between the endpoints of the current two candidate line segments.
[0081] It should be noted that the path search does not simply connect the endpoints of two candidate line segments according to the image distance. Instead, it first uses the surface normal change and the local extremum of transmission attenuation to determine the path propagation order, then uses the main extension direction of the candidate line segments to limit the propagation range, and then uses the number of turns and the direction continuity when entering adjacent candidate line segments to determine the endpoint connection path. This makes the path connection process simultaneously constrained by the visible light direction, the depth surface structure, and the ray transmission structure.
[0082] S3.4. Process the intersections in the set of discontinuous candidate lines and merge the endpoint connection paths to determine the local propagation direction, thereby obtaining a continuous broken candidate skeleton with a local propagation direction.
[0083] When there are intersections in the set of discontinuous candidate lines, each candidate line segment connected by the intersection is treated as an independent branch. The branch propagation direction is determined from the intersection to the endpoint of the corresponding candidate line segment along each candidate line segment. The corresponding endpoint connection path is searched from the endpoint of each candidate line segment according to the processing methods of S3.1 to S3.3.
[0084] The candidate line segments retained in the discontinuous candidate line set are continuously connected with the endpoint connection paths according to the pixel connection relationship; when the endpoint connection paths of different branches pass through the same pixel position, the common path that is repeatedly passed is only retained once, and the pixel connection relationship between the common path and the candidate line segments of each branch is maintained to obtain a continuous broken candidate skeleton.
[0085] For non-intersecting skeleton points in a continuously damaged candidate skeleton, connect the two adjacent skeleton points before and after the skeleton point, and determine the local propagation direction of the corresponding skeleton point by the direction of the line connecting the adjacent skeleton points before and after the skeleton point; for the endpoints of the continuously damaged candidate skeleton, determine the local propagation direction according to the direction of the line connecting the endpoint and the adjacent skeleton point; for intersections, determine the local propagation direction of the corresponding branch according to each branch connected by the intersection.
[0086] By maintaining the correspondence between the local propagation direction and the corresponding skeleton points of the continuous damage candidate skeleton point by point, a continuous damage candidate skeleton with a local propagation direction is obtained.
[0087] S4. Establish image cut-off lines along the local propagation direction of the continuously damaged candidate skeleton, and filter skeleton segments by combining surface structure changes and transmission structure continuity to obtain the material cracked and damaged skeleton and surface deformation area.
[0088] S4.1. Determine the local propagation direction of the skeleton points sequentially along the continuously damaged candidate skeleton, and establish image cut-out lines perpendicular to the corresponding local propagation direction through each skeleton point to obtain the skeleton cut-out line sequence, specifically: Based on the point-to-point correspondence between skeleton points and local propagation directions in the continuously damaged candidate skeleton, skeleton points are selected sequentially along the pixel connection order of the continuously damaged candidate skeleton. Taking the current skeleton point as the location of the image cut-out line, a direction perpendicular to the local propagation direction of the current skeleton point is determined in the image plane of the registered image group, and the direction is extended from the current skeleton point to both sides to the boundary of the enclosed main body region, forming the image cut-out line corresponding to the current skeleton point.
[0089] When the local propagation direction of the continuously damaged candidate skeleton is horizontal, vertical, and two diagonal directions, the image cut-out lines extend along the vertical, horizontal, and corresponding diagonal directions, respectively. When the continuously damaged candidate skeleton is divided into multiple branches at the intersection point, image cut-out lines are established according to the local propagation direction of each branch. The local propagation directions of different branches are not merged. The positional relationship between each image cut-out line and the corresponding skeleton point is saved according to the connection order of the skeleton points to obtain the skeleton cut-out line sequence.
[0090] It should be noted that the image cut-off direction is directly determined by the local propagation direction of the continuously damaged candidate skeleton. Therefore, subsequent changes in surface structure and transmission attenuation are compared along the normal of the damage extension direction to avoid mistaking the continuous changes of the damage itself as differences in the structure on both sides of the skeleton when comparing along the skeleton extension direction.
[0091] S4.2. Based on the pixel mapping relationship, determine the linear texture changes, surface normal changes, and transmission attenuation changes at the corresponding positions of the skeleton cut-off sequence to obtain the skeleton cut-off structural features, specifically: For each image cutoff line in the skeleton cutoff line sequence, taking the corresponding skeleton point as the center, determine the positions of the visible light image pixels adjacent to both sides of the skeleton point along the image cutoff line, and obtain the S-directional texture response. Select the texture response in the corresponding direction according to the local propagation direction of the current skeleton point, and compare the texture response of the skeleton point position with the texture response of the adjacent positions on both sides of the image cutoff line; when the texture response of the skeleton point position is not less than the texture response of the adjacent positions on both sides of the image cutoff line, and is greater than the texture response of one of the adjacent positions, the current skeleton point is determined to have a continuous linear texture discontinuity; if the above relationship is not satisfied, the current skeleton point is determined to not have a continuous linear texture discontinuity.
[0092] Based on the pixel mapping relationship, the current skeleton point and its adjacent positions on both sides of the image cutoff line are mapped to the wrapping depth image. The surface normal change of the current skeleton point and its adjacent positions on both sides of the image cutoff line are determined using the calculation methods of unit surface normal and surface normal change. When the surface normal change of the current skeleton point is simultaneously greater than the surface normal change of its adjacent positions on both sides of the image cutoff line, the current skeleton point is determined to have surface normal change; if the aforementioned local maximum relationship is not satisfied, the current skeleton point is determined to have continuous surface normal.
[0093] Based on the pixel mapping relationship, the same skeleton point and its adjacent positions on both sides of the image cutoff line are mapped to the encapsulated transmission image. The transmission attenuation change is calculated using a method that calculates the transmission attenuation change for both the current skeleton point and its adjacent positions on both sides of the image cutoff line. When the transmission attenuation change for the current skeleton point is simultaneously greater than the transmission attenuation change for its adjacent positions on both sides of the image cutoff line, the current skeleton point is determined to have a transmission attenuation discontinuity; if the aforementioned local maximum relationship is not satisfied, the current skeleton point is determined to have a continuous transmission structure.
[0094] By maintaining the corresponding continuous linear texture discontinuity state, surface normal variation state, and transmission attenuation variation state for each skeleton point according to the connection order of the skeleton points, the skeleton cut-off structure feature is obtained.
[0095] It should be noted that the skeleton cut-off structural features continue to use the already determined multi-directional texture response, surface normal variation, and transmission attenuation variation, without establishing new judgment parameters; all three types of variations are limited to the same spatial location of the wrapping surface by pixel mapping relationship, so that visible light texture variation, depth surface variation, and ray transmission variation have a consistent spatial comparison benchmark.
[0096] S4.3. Based on the structural characteristics of the skeleton cross-section, the skeleton segments corresponding to the material cracks are selected to obtain the material cracked and damaged skeleton, specifically as follows: The structural features of the skeleton cut-off line are checked point by point according to the pixel connection order of the continuous broken candidate skeleton. When the skeleton point has continuous linear texture discontinuity, transmission attenuation discontinuity and surface normal change at the same time, the corresponding skeleton point is determined as the material crack skeleton point. The adjacent skeleton points are checked along the continuous broken candidate skeleton. The adjacent and continuous material crack skeleton points that meet the above three structural change relationships are connected according to the original pixel connection order to form the material crack skeleton segment.
[0097] For each intersection point, the structural features of the corresponding skeleton section are examined according to the local propagation direction of each branch connected by the intersection point. Only material cracking skeleton points that continuously satisfy the three structural change relationships within the same branch are connected, and material cracking skeleton points are not connected across different branches. All material cracking skeleton segments are preserved according to the original connection relationship of the continuous failure candidate skeleton, resulting in a material cracking failure skeleton with a local propagation direction.
[0098] When the skeleton point has only continuous linear texture discontinuity, while the surface normal remains continuous and the transmission structure is continuous, the corresponding skeleton point is deleted; when the aforementioned relationship occurs continuously, the corresponding continuous skeleton segment is deleted; the deleted skeleton segment does not enter the material cracking and damage skeleton and subsequent surface deformation area.
[0099] It should be noted that the material cracking and damage skeleton is not determined solely by the linear edges in the visible light image, but also requires the presence of surface normal changes and transmission attenuation discontinuities at the same spatial location. Therefore, even if printed lines, flat tape edges, and illumination boundaries form continuous linear responses in the visible light image, they are still excluded from the continuous damage candidate skeleton when the depth surface and the X-ray transmission structure remain continuous.
[0100] S4.4. Based on the structural characteristics of the skeleton cross-section, determine the region where the transmission structure is continuous and the depth surface forms a continuous folding deformation along the continuous skeleton points to the same side, thus obtaining the surface deformation region, specifically: For continuous broken candidate skeletons that have not been identified as material cracking skeleton points, continue to check the skeleton cross-section structural features according to the skeleton point connection order; when the transmission structure of the current skeleton point is continuous and there is a change in surface normal, compare the three-dimensional surface coordinates of adjacent depth pixels on both sides of the skeleton point along the image cross-section corresponding to the current skeleton point, determine the side where the surface depth changes, and take the side where the surface depth changes as the turning direction of the current skeleton point.
[0101] The adjacent skeleton points are checked sequentially along the continuous broken candidate skeleton. When the adjacent skeleton points maintain the continuity of the transmission structure, have surface normal changes, and the turning direction of each skeleton point is on the same side of the corresponding image cut-off line, the adjacent skeleton points are connected continuously to obtain a continuous turning skeleton segment. When the turning direction of the adjacent skeleton points changes, the current continuous turning skeleton segment is terminated at the position where the turning direction changes, and the turning direction is rechecked from the subsequent skeleton points.
[0102] For continuous folded skeleton segments, the range of continuous depth pixels where the surface normal changes is determined along the image cut-off lines of each skeleton point, and the range of continuous depth pixels corresponding to adjacent skeleton points is connected according to the pixel adjacency relationship in the depth image to form the surface deformation region; for intersection points, the image cut-off lines and folding directions are determined according to the local propagation direction of each branch, and only the range of continuous depth pixels with consistent folding directions within the same branch is connected.
[0103] It should be noted that the surface deformation area requires the X-ray transmission structure to remain continuous, while the depth surface is required to remain folded on the same side along the continuous skeleton points. Therefore, the surface deformation area and the material cracking and damage skeleton adopt different structural judgment relationships. The material cracking and damage skeleton corresponds to the position where the visible light texture, depth surface and X-ray transmission structure change simultaneously, while the surface deformation area corresponds to the position where the X-ray transmission structure is continuous and the depth surface continues to fold.
[0104] S5. Establish orthogonal boundary search lines along the material crack and damage skeleton, and correct paired boundary points according to the boundary extension direction. Combine the surface deformation area and pixel mapping relationship to determine the damage area and damage scale, and generate package damage detection results.
[0105] S5.1. Divide the skeleton into non-intersecting segments according to the pixel connectivity of the material cracking and damage skeleton, and determine the local propagation direction based on the positional relationship between the current skeleton point and the adjacent skeleton points in each segment. Extend the search lines to both sides of the skeleton along a direction perpendicular to the corresponding local propagation direction, with each skeleton point as the center, to obtain the orthogonal boundary search lines, specifically: The eight-neighborhood connectivity of the material cracking and damage skeleton is examined pixel-by-pixel. Intersections are used as boundaries between different skeleton branches. Skeleton points continuously connected from one intersection to the next intersection or between skeleton endpoints are divided into non-intersecting skeleton segments. For each non-intersecting skeleton segment, the current skeleton point is selected sequentially according to the connection order. Non-endpoint positions are connected to the skeleton points immediately before and after the current skeleton point, and the direction of the line connecting these adjacent skeleton points is used as the local propagation direction of the current skeleton point. The local propagation direction for skeleton segment endpoints is the direction of the line connecting the endpoint and its adjacent skeleton point.
[0106] Using the current skeleton point as the center of the orthogonal boundary search line, a direction perpendicular to the local propagation direction of the current skeleton point is selected within the image plane encompassing the visible light image. This direction extends pixel by pixel from the current skeleton point to the boundary of the enclosing main body region, forming the orthogonal boundary search line corresponding to the current skeleton point. When the local propagation direction is horizontal, the orthogonal boundary search line extends vertically; when the local propagation direction is vertical, the orthogonal boundary search line extends horizontally; when the local propagation direction is 45° diagonal, the orthogonal boundary search line extends 135° diagonally; when the local propagation direction is 135° diagonal, the orthogonal boundary search line extends 45° diagonally.
[0107] For different branches connected by intersection points, orthogonal boundary search lines are established using the local propagation direction corresponding to each branch, without using a single orthogonal boundary search line to process different branches simultaneously. The positional relationship between each skeleton point and its corresponding orthogonal boundary search line is saved according to the connection order of the skeleton points, resulting in orthogonal boundary search lines continuously distributed along the material cracking and damage skeleton.
[0108] It should be noted that the orthogonal boundary search line is always perpendicular to the local propagation direction of the material cracking and damage skeleton, so that the subsequent boundary search is carried out along the damage width direction, rather than along the extension direction of the material cracking and damage skeleton itself.
[0109] S5.2. Search along the orthogonal boundary search line from the corresponding skeleton point to both sides for the nearest edge position in the visible light image, obtain paired boundary points, and determine the extension direction of the two sides of the boundary according to the skeleton propagation order, specifically: For each skeleton point, an orthogonal boundary search line is searched pixel-by-pixel along both opposite directions of the orthogonal boundary search line, starting from the skeleton point. During the search, the obtained multi-directional texture response and non-maximum suppression results are used to check whether each pixel position on the orthogonal boundary search line belongs to a retained single-pixel line. The position of the first single-pixel line encountered along one side of the orthogonal boundary search line is determined as the corresponding side boundary point, and the position of the first single-pixel line encountered along the other side is determined as the other side boundary point. The two boundary points obtained on both sides of the same skeleton point are determined as a pair of boundary points.
[0110] When the orthogonal boundary search line extends to the boundary of the main body area but still does not encounter a single pixel line, the corresponding side is marked as a visible light boundary interruption and no corresponding side boundary point is formed for the time being.
[0111] Arrange the paired boundary points corresponding to each skeleton point in sequence according to the local propagation direction of the material cracking and damage skeleton. Connect the boundary points of adjacent skeleton points on the same side according to the skeleton propagation sequence. The direction of the line connecting two adjacent boundary points on the same side is used as the boundary extension direction of the corresponding segment.
[0112] Since the horizontal, 45° diagonal, vertical and 135° diagonal directions have been used to describe the direction of linear texture and skeleton propagation, the direction of the line connecting adjacent boundary points on the same side is compared with the above four directions respectively, and the direction with the smallest included angle is determined as the corresponding boundary extension direction, so that the boundary extension direction and the local propagation direction of material cracking and damage skeleton adopt the same direction representation method.
[0113] S5.3. Compare the extension directions of the two side boundaries with the local propagation directions of the corresponding skeleton points. For boundary points with inconsistent directions, move them along the corresponding boundary to obtain the corrected paired boundary points, specifically: The boundary extension directions on both sides of the current skeleton point are compared with the local propagation direction of the current skeleton point. When the boundary extension direction on one side is consistent with the local propagation direction of the current skeleton point, the corresponding side boundary point is retained; when the boundary extension direction on one side is inconsistent with the local propagation direction of the current skeleton point, the corresponding side boundary point is not directly used as the damaged boundary.
[0114] For boundary points with inconsistent extension directions, the system moves pixel-by-pixel along the boundary formed by connecting adjacent boundary points on the corresponding side towards the skeleton propagation direction. After each move, the moved boundary point is reconnected with the corresponding boundary point on the same side of the adjacent skeleton point to redetermine the boundary extension direction. When the redetermined boundary extension direction matches the local propagation direction of the current skeleton point, the movement stops, and the current position is taken as the corrected boundary point. If, after moving along the corresponding boundary to the corresponding position of the adjacent skeleton point, a boundary position with a consistent direction is still not obtained, the existing boundary position closest to the current skeleton point and whose boundary extension direction matches the local propagation direction is retained, and the search continues without crossing adjacent skeleton points.
[0115] After correcting the boundary points on both sides of the current skeleton point, keep the corrected boundary points on both sides of the same orthogonal boundary search line paired to obtain the corrected paired boundary points corresponding to the current skeleton point.
[0116] For a curved skeleton, the orthogonal boundary search line is updated point by point according to the local propagation direction that has been determined for the continuous skeleton points. The boundary search on both sides and the boundary extension direction comparison are re-executed on the updated orthogonal boundary search line so that the corrected paired boundary points change continuously with the orientation of the material cracking and broken skeleton.
[0117] For bifurcated skeletons, each non-intersecting skeleton segment is entered with the intersection point as the boundary. Orthogonal boundary search lines are established using the local propagation direction corresponding to each branch. Paired boundary point search and boundary extension direction correction are performed separately, and boundary points on different branches are not grouped into the same pair of boundary points.
[0118] It should be noted that the material cracking and damage skeleton only determines the path of the damage center. The initial edge position obtained by the orthogonal boundary search may still come from the section adjacent to the bending crack. After relocating the boundary points by using the consistent relationship between the boundary extension direction and the local propagation direction, the paired boundary points correspond to the damage boundaries on both sides of the same skeleton position, rather than the adjacent edges of different skeleton sections.
[0119] S5.4. When the visible light boundary is interrupted on one side, the candidate boundary position is determined along the surface normal abrupt change line at the corresponding position, and the candidate boundary position is verified by the continuous transmission attenuation change in the wrapped transmission image, and the corresponding side boundary point is completed, specifically: For orthogonal boundary search lines for which one side boundary point has not been obtained, the orthogonal boundary search lines are mapped to the wrapping depth image according to the pixel mapping relationship. Using the obtained surface normal change results, the surface normal change is checked pixel by pixel from the current skeleton point outward along one side of the missing boundary. The position where the surface normal change first occurs is determined as the candidate boundary position.
[0120] The candidate boundary positions are mapped to the packaged transmission image according to the pixel mapping relationship, and the transmission attenuation changes of the corresponding positions of adjacent skeleton points along the local propagation direction of the material cracking and damage skeleton are checked. When there are transmission attenuation changes at both the candidate boundary position and the candidate boundary positions corresponding to the adjacent skeleton points, and the adjacent candidate boundary positions can be continuously connected according to the eight-neighbor relationship, it is determined that the candidate boundary position has a continuous transmission attenuation change along the propagation direction of the material cracking and damage skeleton, and the verification is passed. The candidate boundary position is then used as the corresponding side boundary point.
[0121] If a candidate boundary position cannot form an eight-neighbor contiguous connection with an adjacent candidate boundary position, or if there is no change in transmission attenuation at the corresponding package transmission image position, the verification fails. The candidate boundary position is not used to complete the corresponding side boundary point, and the search continues outward along the orthogonal boundary search line to check for the next surface normal abrupt change position. If no verified candidate boundary position is found when the orthogonal boundary search line reaches the boundary of the package's main body region, paired boundary points for the corresponding skeleton points are not formed.
[0122] Pair the verified corresponding boundary points with the determined boundary points on the other side, and then correct the boundary positions according to the boundary extension direction comparison method to obtain the corrected paired boundary points.
[0123] S5.5. Connect the corrected paired boundary points sequentially according to the local propagation direction of the material cracking and damage skeleton, and close the boundary to obtain the material cracking and damage area; complete the region closure along the continuous folding boundary of the surface deformation area to obtain the surface deformation and damage area, specifically: For each non-intersecting skeleton segment, the corrected paired boundary points are arranged according to the local propagation direction of the material cracking and damage skeleton, and the adjacent corrected boundary points on the same side are connected to form continuous boundaries on both sides of the material cracking and damage skeleton. At the beginning position of the non-intersecting skeleton segment, the two corrected boundary points corresponding to the starting skeleton point are connected, and at the end position of the non-intersecting skeleton segment, the two corrected boundary points corresponding to the ending skeleton point are connected to close the continuous boundaries on both sides, thus obtaining the material cracking and damage area of the corresponding non-intersecting skeleton segment.
[0124] For the bifurcated skeleton, material cracking and damage areas are formed according to each non-intersecting skeleton segment, and adjacent material cracking and damage areas are connected according to common boundary pixels at the common intersection position, while preserving the original boundary direction of each branch.
[0125] For the surface deformation region, the continuous depth pixels within the surface deformation region are checked according to the eight-neighborhood relationship. The outermost continuous depth pixels that have adjacent pixels inside the surface deformation region and are adjacent to the pixels outside the surface deformation region are determined as the continuous turning boundary. The adjacent boundary pixels are connected sequentially along the continuous turning boundary and returned to the starting boundary pixel to complete the region closure and obtain the surface deformation damage region. The material cracking damage region and the surface deformation damage region together serve as the encapsulation damage region.
[0126] S5.6. By using pixel mapping relationships, the damaged area and boundary of the package are transformed to the actual spatial location, the damage scale is determined, and the package damage detection result is generated, specifically as follows: According to the pixel mapping relationship, the corrected paired boundary points in the cracked and damaged area of the material are respectively mapped to the pixel positions of the depth image, and the actual spatial positions of the corrected paired boundary points are determined by the three-dimensional surface coordinates; in the same way, the boundaries of the cracked and damaged area of the material and the boundaries of the surface deformation and damaged area are mapped to the actual spatial positions.
[0127] For a pair of corrected boundary points corresponding to the same skeleton point, the distance between the two actual spatial locations is calculated using spatial Euclidean distance, and the calculation result is determined as the local damage width of the corresponding skeleton point.
[0128] Spatial Euclidean distance directly adopts the distance relationship between two points in a three-dimensional rectangular coordinate system, and the expression is: ; In the formula, Indicates the first The width of the local damage corresponding to each skeleton point; , and Indicates the first The three-dimensional spatial coordinates of the boundary point after correction on one side of the skeleton point; , and Indicates the first The three-dimensional spatial coordinates of the boundary point on the other side of the skeleton point after correction.
[0129] Calculate the local damage width of all pairs of boundary points on the cracked skeleton of the material in sequence, and select the local damage width with the largest value as the maximum damage width.
[0130] For the boundary of the material cracked and damaged area transformed to the actual spatial location, adjacent spatial boundary points are determined sequentially according to the boundary pixel connection order, and the distance between adjacent spatial boundary points is calculated using spatial Euclidean distance. The distances of all adjacent spatial boundary points are summed to determine the length of the damaged boundary, expressed as: ; In the formula, Indicates the length of the damaged boundary; Indicates the number of spatial boundary points contained within the closed broken boundary; , and Indicates the first The three-dimensional spatial coordinates of the spatial boundary points; , and Indicates the relationship with the first The three-dimensional spatial coordinates of the next spatial boundary point adjacent to the first spatial boundary point; , and Indicates the first The three-dimensional spatial coordinates of the spatial boundary points.
[0131] For the surface deformation and damage area, the three-dimensional surface coordinates on both sides of the continuous turning boundary are determined along each established image cut-off line, and the spatial straight line between the three-dimensional surface coordinates of the two boundaries is used as the local surface reference of the corresponding image cut-off line.
[0132] For each 3D surface coordinate located between two consecutive turning boundary points, calculate the spatial distance from the 3D surface coordinate to the local surface reference.
[0133] The distance from a point in space to a line adopts the point-to-line distance relationship in three-dimensional analytical geometry, and the expression is: ; In the formula, Indicates the first The deformation depth of a three-dimensional surface coordinate relative to a corresponding local surface reference; Indicates the first [item] within the surface deformation and damage area. Three-dimensional surface coordinates; and They respectively represent the first The coordinates of two consecutive folded boundary 3D surface coordinates lie on the same image cutoff line.
[0134] The deformation depth of each effective three-dimensional surface coordinate within the surface deformation and damage area is calculated sequentially, and the deformation depth with the largest value is selected as the maximum surface deformation depth.
[0135] The actual spatial location of the damaged area of the package, the boundary of the material cracked area, the boundary of the surface deformed area, the length of the damaged boundary, the maximum width of the damage, and the maximum surface deformation depth are recorded accordingly to generate the package damage detection results.
[0136] In summary, this invention gradually restores the package damage from a discrete linear response to a continuous candidate skeleton with a local propagation direction, ensuring that local missing, weak response, and intersection areas maintain a unified damage extension relationship and preventing the detection results from being fragmented due to local image response interruptions. Furthermore, by utilizing the surface normal changes and transmission attenuation changes on both sides of the skeleton, the nature of the damage can be confirmed, further distinguishing between material cracking and simple surface folding, allowing different damage morphologies to correspond to their respective geometric regions. For material cracked areas, an orthogonal boundary search relationship is established along the local propagation direction of the skeleton, and the positions of the two sides of the boundary are corrected by combining the boundary's own extension direction. This ensures that curved sections, bifurcation sections, and local boundary interruption positions still maintain a relatively stable left-right boundary correspondence, reducing width distortion caused by directly taking the nearest edge. By relying on pixel mapping relationships to restore the image boundary to the actual surface space of the package, the damaged area, maximum damage width, damage boundary length, and maximum surface deformation depth originate from the same spatial geometric relationship, thereby improving the consistency of package damage detection results in terms of location, shape, and scale.
[0137] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for detecting package damage based on image extraction, characterized in that: include, The visible light image, depth image, and ray transmission image of the package to be detected are acquired, spatial registration is performed, and the main body region of the package is extracted to obtain the registered package image group and pixel mapping relationship; Linear textures are extracted from the registered image group, and the linear textures are filtered by combining surface normal variation and transmission attenuation variation to obtain a set of discontinuous candidate lines with the main extension direction. The path search area is determined based on the main extension direction of the discontinuous candidate line set. The minimum path search is performed by combining the surface normal change and the local extreme value of transmission attenuation, and the candidate line segments are connected to obtain a continuous broken candidate skeleton with a local propagation direction. Image cut-outs are established along the local propagation direction of the continuous broken candidate skeleton. Skeleton segments are screened by combining surface structure changes and transmission structure continuity to obtain the material cracked and broken skeleton and surface deformation area. An orthogonal boundary search line is established along the cracked and damaged skeleton of the material, and the paired boundary points are corrected according to the boundary extension direction. The damaged area and damage scale are determined by combining the surface deformation area and pixel mapping relationship, and the package damage detection results are generated.
2. The package damage detection method based on image extraction as described in claim 1, characterized in that: The acquisition of the registration-wrapped image group and pixel mapping relationship specifically includes: The visible light image and depth image are transformed according to the calibration relationship of the imaging device to obtain a spatially aligned visible light image and depth image; Based on the spatial positional relationship between the packaged surface and the bearing reference surface in the spatially aligned depth image, the packaged main body region is separated, and the depth pixels within the packaged main body region are converted into surface spatial positions; The surface spatial position is projected onto the X-ray transmission image, and the corresponding image area is cropped according to the main body area of the package to obtain the registered package image group; Pixel correspondences between visible light images, depth images, and ray transmission images are established based on surface spatial location and projection location, forming pixel mapping relationships.
3. The package damage detection method based on image extraction as described in claim 2, characterized in that: The extraction of linear textures from the registered and enclosed image group specifically includes: The visible light images in the registered image group are converted into grayscale images. Gabor filtering is then applied to the grayscale images along the horizontal, vertical, and two diagonal directions to obtain multi-directional texture responses. By comparing the multi-directional texture responses at the same pixel location, retaining the direction with the most obvious response, determining the pixel texture direction, and performing non-maximum suppression along the direction perpendicular to the pixel texture direction, the texture response with width is refined into a single pixel line, resulting in a linear texture.
4. The package damage detection method based on image extraction as described in claim 3, characterized in that: The acquisition of the discontinuous candidate line set specifically includes: The corresponding positions of the linear texture in the depth image and the ray transmission image are determined according to the pixel mapping relationship. The surface normal changes of the neighborhood on both sides of the linear texture are calculated, and the transmission attenuation changes along the normal direction of the linear texture are extracted. Remove linear textures where the surface normal and transmission attenuation remain continuous, and retain line segments where the surface normal and transmission attenuation change to obtain candidate line segments. The endpoints and intersections are determined according to the pixel connectivity of the candidate line segments, and the main extension direction is determined along the continuous direction of the candidate line segments, resulting in a set of discontinuous candidate lines with the main extension direction.
5. The package damage detection method based on image extraction as described in claim 4, characterized in that: The determination of the path search region based on the main extension direction of the discontinuous candidate line set specifically includes: Select the endpoints of candidate line segments from the set of discontinuous candidate lines, and determine the search direction in front of the endpoints according to the main extension direction of the corresponding candidate line segments; Expand the search range of the candidate line segment endpoints along the search direction, retaining adjacent pixels that are continuous with the main extension trend of the candidate line segment, to obtain the path search area.
6. The package damage detection method based on image extraction as described in claim 5, characterized in that: The acquisition of the continuously damaged candidate skeleton specifically includes: The surface normal abrupt change and transmission attenuation local extrema at the corresponding positions of each path node in the path search area are detected according to the pixel mapping relationship. Path nodes with two structural changes are determined as the first passable node, path nodes with only one structural change are determined as the second passable node, and path nodes that only maintain visible light direction response are determined as the third passable node. The path propagation order is determined according to the order of the first passable node, the second passable node, and the third passable node. The path is extended step by step from the endpoint of the current candidate line segment to the endpoint of the adjacent candidate line segment according to the path propagation order, and the path is restricted to propagate along the main extension trend of the current candidate line segment, so as to obtain the reachable path connecting the current candidate line segment and the adjacent candidate line segment. When there are multiple reachable paths between two candidate line segments, determine the number of turns and the propagation direction when entering the adjacent candidate line segment for each reachable path. Retain the reachable path with the fewest turns and the propagation direction that is consistent with the main extension direction of the adjacent candidate line segment to obtain the endpoint connection path. When there are intersections in the set of discontinuous candidate lines, the endpoint connection paths are extended from the intersections to the endpoints of the corresponding candidate line segments, the common paths that are repeatedly passed are merged, and the local propagation direction is determined to obtain a continuous broken candidate skeleton with a local propagation direction.
7. The package damage detection method based on image extraction as described in claim 6, characterized in that: The acquisition of the material's cracked and damaged skeleton and surface deformation area specifically includes: The local propagation direction of the skeleton points is determined sequentially along the continuously damaged candidate skeleton. Image cut-outs perpendicular to the corresponding local propagation direction are established through each skeleton point to obtain the skeleton cut-out sequence. Based on the pixel mapping relationship, determine the linear texture change, surface normal change and transmission attenuation change at the corresponding position of the skeleton cut-off sequence to obtain the skeleton cut-off structure features. Retain the skeleton segments that simultaneously have continuous linear texture discontinuity, transmission attenuation discontinuity and surface normal change to obtain the material cracking and damage skeleton. Based on the structural characteristics of the skeleton cross-section, the area where the continuous transmission structure and the depth surface form a continuous folding deformation along the continuous skeleton points to the same side is defined as the surface deformation area.
8. The package damage detection method based on image extraction as described in claim 7, characterized in that: The establishment of orthogonal boundary search lines specifically includes: The skeleton segments without intersections are divided according to the pixel connectivity of the material cracking and damage skeleton, and the local propagation direction is determined according to the positional relationship between the current skeleton point and the adjacent skeleton points in each skeleton segment. With each skeleton point as the center, extend the search lines to both sides of the skeleton in a direction perpendicular to the corresponding local propagation direction to obtain orthogonal boundary search lines.
9. The package damage detection method based on image extraction as described in claim 8, characterized in that: The step of correcting the paired boundary points according to the boundary extension direction specifically includes: Search for the nearest edge position in the visible light image from the corresponding skeleton point to both sides along the orthogonal boundary search line to obtain paired boundary points; Select paired boundary points corresponding to adjacent skeleton points according to the skeleton propagation order, connect adjacent boundary points on the same side respectively, and determine the extension direction of the two sides of the boundary. Compare the extension directions of the two sides of the boundary with the local propagation direction of the corresponding skeleton point. For boundary points with inconsistent directions, move them to the front and back sides of the corresponding boundary towards the skeleton propagation direction, and retain the boundary positions where the boundary extension direction is consistent with the local propagation direction. For curved skeletons, the orthogonal boundary search line is updated according to the local propagation direction of continuous skeleton points, and the paired boundary points are continuously corrected according to the boundary extension direction. For bifurcated skeletons, the corresponding paired boundary points are corrected along the local propagation direction of each branch skeleton. When the visible light boundary is interrupted on one side, the candidate boundary position is determined along the surface normal abrupt change line at the corresponding position. The candidate boundary position is then verified using the continuous transmission attenuation change in the X-ray transmission image. The verified candidate boundary position is then used to fill in the corresponding side boundary point to obtain the corrected paired boundary point.
10. The package damage detection method based on image extraction as described in claim 9, characterized in that: The generation of package damage detection results specifically includes: By sequentially connecting the corrected paired boundary points and closing the boundary according to the local propagation direction of the material cracking and damage skeleton, the material cracking and damage area is obtained. The region is closed by continuously turning the boundary along the surface deformation region, thus obtaining the surface deformation and damage region. The damaged area of the package is determined by combining the cracked and damaged areas of the material and the deformed and damaged areas of the surface, and the damaged area and boundary of the package are transformed to the actual spatial location through pixel mapping relationship; The local damage width is determined based on the converted paired boundary points, and the maximum damage width is extracted. The damage boundary length is determined along the converted damage boundary, and the maximum surface deformation depth is determined in combination with the surface deformation damage area to generate the package damage detection result.
