Target laser stripe center line extraction method, system, device and medium based on visual simulation prior information

Abstract

The application discloses a target laser stripe center line extraction method, system, device and medium based on visual simulation prior information, and the method comprises the following steps: S1, laser stripe images are collected by collecting a to-be-measured scene; S2, visual simulation is performed on the to-be-measured scene to obtain simulation laser stripes of target laser stripes; S3, a region of interest is determined, and the camera collected image is cropped according to the region of interest to obtain a cropped laser stripe image; S4, a direction parameter is calculated according to the direction information of the simulation laser stripes, and a Gabor filter is established; S5, the cropped laser stripe image is filtered by using the Gabor filter, laser stripes with a specific direction in the image are detected, and target laser stripes are obtained; and S6, a Steger algorithm is used for laser stripe center line extraction to obtain target laser stripe center lines. The prior information provided by visual simulation is used in laser stripe recognition and extraction, laser stripes in the image can be classified and target stripes can be extracted.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method, system, device, and medium for extracting the center line of a target laser stripe based on visual simulation prior information. Background Technology

[0002] Structured line light sensors are among the most common active vision sensors, boasting advantages such as strong anti-interference capabilities, simple structure, and high measurement accuracy, leading to their widespread application in industrial fields. A structured line light sensor projects laser stripes onto the surface of an object using a laser, which are then captured by a camera to obtain an image of the laser stripes. The laser stripes change according to the shape of the object's surface; by processing and analyzing the laser stripe image, the object can be measured.

[0003] Traditional laser stripe center extraction algorithms include edge methods, center methods, thresholding methods, thinning methods, grayscale centroid methods, and the Steger algorithm. Among these, edge methods, center methods, and thresholding methods are simple to operate and fast, but their accuracy is poor, making them only suitable for rough estimation of the center line. Thinning methods, also known as morphological skeleton methods, are morphological processing methods that repeatedly erode the light stripes, stripping away the stripe boundaries to obtain a single-pixel-width connected line of the light stripes (also called the skeleton). This repetitive operation leads to low computational speed. Grayscale centroid methods and the simple Steger algorithm are easily affected by noise, and the Steger algorithm has a large computational load, making it unable to achieve real-time results. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and to propose a method, system, device and medium for extracting the center line of a target laser stripe based on visual simulation prior information. The method distinguishes stripes from different surfaces of an object on a laser stripe image and extracts the center line of the target laser stripe.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The method for extracting the center line of a target laser stripe based on prior information from visual simulation includes the following steps:

[0007] S1. Use a line structured light sensor to collect data on the scene under test to obtain a laser stripe image;

[0008] S2. Perform visual simulation on the scene to be tested to obtain the simulated laser stripe of the target laser stripe in the current state;

[0009] S3. Determine the region of interest based on the distribution range of the simulated laser stripes in the camera-acquired image, and crop the camera-acquired image based on the region of interest to obtain the cropped laser stripe image.

[0010] S4. Calculate the direction parameters based on the obtained direction information of the simulated laser stripes and establish a two-dimensional Gabor filter;

[0011] S5. Filter the cropped laser stripe image using a two-dimensional Gabor filter to detect laser stripes with specific directions in the image and obtain the target laser stripe.

[0012] S6. Use the Steger algorithm to extract the center line of the laser stripe from the obtained target laser stripe image to obtain the center line of the target laser stripe.

[0013] This invention also includes a target laser stripe centerline extraction system based on visual simulation prior information. The system applies the target laser stripe centerline extraction method provided by this invention. The system includes:

[0014] Image acquisition module, visual simulation module, image processing module, filtering module, and laser stripe center line extraction module;

[0015] The image acquisition module is used to acquire images of the scene under test and obtain laser stripe images;

[0016] The visual simulation module is used to perform visual simulation of the scene under test and obtain the simulated laser stripe of the target laser stripe in the current state.

[0017] The image processing module is used to determine the region of interest based on the distribution range of the simulated laser stripes in the camera-acquired image, and to crop the camera-acquired image based on the region of interest to obtain the cropped laser stripe image.

[0018] The filtering module calculates the direction parameters based on the direction information of the obtained simulated laser stripes, establishes a two-dimensional Gabor filter, and uses the two-dimensional Gabor filter to filter the cropped laser stripe image, detects laser stripes with specific directions in the image, and obtains the target laser stripe.

[0019] The laser stripe centerline extraction module is used to extract the laser stripe centerline from the obtained target laser stripe image using the Steger algorithm.

[0020] The present invention also includes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the target laser stripe centerline extraction method provided by the present invention.

[0021] The present invention also includes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the target laser stripe centerline extraction method provided by the present invention.

[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0023] 1. This invention uses prior information of simulated laser stripes generated by visual simulation, which enables the invention to distinguish laser stripes from different surfaces; the invention achieves classification in the image preprocessing stage, reducing the amount of computation and further improving real-time performance.

[0024] 2. By combining prior information from visual simulation to delineate the region of interest, interference factors can be eliminated as much as possible. Since the size of the part that needs to be processed is reduced, the speed of subsequent image processing can be accelerated. Compared with the general method of dividing the region of interest, using the distribution of simulated laser stripes in the image to determine the region of interest is closer to the target area.

[0025] 3. In images, laser stripes on different surfaces often have different shapes and directions, which can be used to distinguish laser stripes. This invention extracts direction information from simulated laser stripes and then uses a Gabor filter to process the actual image, which can extract laser stripes with specific directions, making it more targeted.

[0026] 4. Finally, the present invention uses the Steger algorithm to extract the center line, which can obtain the center of the laser stripe at the sub-pixel level. It has the advantages of high accuracy and good robustness of the Steger algorithm. At the same time, since the image is processed with reference to simulation information, the amount of computation is greatly reduced, overcoming the disadvantages of Steger's large amount of computation and poor real-time performance. This makes the center line extraction method applicable to high real-time processes such as welding. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method of the present invention;

[0028] Figure 2 It is an actual laser stripe image acquired by a line structured light sensor;

[0029] Figure 3 It is a simulated laser stripe image of the target obtained through visual simulation;

[0030] Figure 4 This is a schematic diagram of the region of interest;

[0031] Figure 5 This is a graph showing the Gabor filtering results;

[0032] Figure 6 This is the result of extracting the center line of the laser stripe. Detailed Implementation

[0033] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0034] Example

[0035] like Figure 1 As shown, the method for extracting the center line of a target laser stripe based on prior information from visual simulation includes the following steps:

[0036] S1. Use a line structured light sensor to collect data on the scene under test to obtain a laser stripe image;

[0037] S2. Perform visual simulation on the scene to be tested to obtain the simulated laser stripe of the target laser stripe in the current state; specifically, reproduce the imaging process of the visual sensor in the digital twin system to obtain the acquisition results in the simulated environment, which, for the line structured light sensor, generates a simulated laser stripe image.

[0038] First, the line structured light sensor is calibrated to obtain the camera's intrinsic and extrinsic parameters, as well as the laser plane equation. The intrinsic parameters are the camera's internal properties, including focal length, principal point coordinates, and distortion coefficients; the extrinsic parameters describe the camera's pose in the world coordinate system, including rotation matrices and translation vectors. A laser plane is established in the simulation environment. Within the laser plane, a set of rays {r1,…,r...} are established at certain angular intervals, centered on the laser emission origin. n The laser plane is discretized into several rays, and the intersection point of each ray with the model is calculated. The intersection line obtained by connecting the intersection points is the laser stripe to be obtained. The number of rays n determines the shape accuracy of the generated simulated laser stripe. The more rays there are, the higher the shape accuracy of the laser stripe.

[0039] Then, for each laser ray r i Find the intersection points of the ray and the workpiece model. The 3D model of the workpiece is usually composed of multiple surfaces, some of which may be curved. Therefore, each ray may have multiple intersection points with the model. Since light cannot penetrate opaque surfaces, only one intersection point of the same ray actually exists. Therefore, the visual simulation algorithm only retains the intersection point p closest to the origin of the laser emission. i ;

[0040] Finally, by connecting the intersections of all the rays, we obtain the laser stripes formed on the surface of the virtual workpiece by the virtual line structured light sensor.

[0041] like Figure 2 As shown, this is an actual laser stripe image acquired by a line structured light sensor. The image contains laser stripes in multiple directions as well as interference such as arc light and reflected light on the workpiece surface. The laser stripe on the right is the target stripe to be extracted.

[0042] like Figure 3As shown, this is a simulated laser stripe image of the target obtained through visual simulation; the simulated laser stripe of the target has a high degree of consistency with the actual stripe to be extracted in terms of position and orientation information.

[0043] S3. Determine the region of interest based on the distribution range of the simulated laser stripes in the camera-acquired image, and crop the camera-acquired image based on the region of interest to obtain the cropped laser stripe image; specifically:

[0044] Since the distribution of simulated laser stripes in the image is basically consistent with that of actual laser stripes, the region of interest (ROI) is determined based on the distribution of simulated laser stripes in the camera-captured image, so that the target stripe is located within the ROI for further processing. The remaining stripes and most of the interference are located outside the ROI and are not processed. Considering the deviation between the actual situation and the simulation, the actual stripes and the simulated stripes will not be completely consistent. Therefore, it is necessary to appropriately expand the range of the ROI to increase the adaptability of the algorithm. At the same time, the size of the image also needs to be considered to ensure that the ROI does not exceed the range of the camera-captured image.

[0045] The formula for calculating the region of interest is as follows:

[0046] x tl =max{0,min{x i}-50}

[0047] y tl =max{0,min{y i}-50}

[0048] x br =min{W,max{x i +50

[0049] y br =min{H,max{y i +50

[0050] Among them, (x tl ,y tl ) and (x br ,y br The values ​​{x1, ..., x2} represent the pixel coordinates of the top-left and bottom-right points of the rectangular region of interest; n} and {y1,...,y n} are the pixel coordinates of the simulated laser stripe points in the x and y directions of the image captured by the camera, and are sorted in ascending order of x coordinate; W and H are the width and height of the image captured by the camera, respectively.

[0051] like Figure 4The diagram shows the region of interest. The region of interest is determined by the position information provided by the simulated laser stripes, and all subsequent operations are performed within this region.

[0052] S4. Calculate the direction parameters based on the obtained direction information of the simulated laser stripes, and establish a two-dimensional Gabor filter; the specific formula for calculating the direction parameters is as follows:

[0053]

[0054] Where n is the number of simulated laser stripe points;

[0055] The two-dimensional Gabor filter is specifically represented as follows:

[0056]

[0057] S5. Filter the cropped laser stripe image using a two-dimensional Gabor filter to detect laser stripes with specific directions in the image and obtain the target laser stripe.

[0058] The formula for the real part of a two-dimensional Gabor filter is:

[0059]

[0060] x′=xcosθ+ysinθ

[0061] y ′ = -xsinθ + ycosθ

[0062] Where (x,y) are the pixel coordinates in the laser stripe image; λ is the wavelength of the filter kernel, which directly affects the scale of the filter kernel; θ is the direction of the filter; ψ is the phase shift of the tuning function; σ is the standard deviation of the Gaussian kernel function; and γ is the aspect ratio of the kernel function, which determines the shape of the filter.

[0063] The cropped laser stripe image is filtered using a two-dimensional Gabor filter and then segmented by thresholding to obtain laser stripes with a specific direction.

[0064] like Figure 5 The image shows the Gabor filtering result; Gabor filtering based on the direction information provided by the simulated laser stripes can extract laser stripes with specific directions.

[0065] S6. Use the Steger algorithm to extract the center line of the laser stripe from the obtained target laser stripe image; specifically:

[0066] The Steger algorithm first applies a Gaussian filter to the target laser stripe image, where the filter scale σ needs to satisfy:

[0067]

[0068] Where ω is the width of the laser stripe;

[0069] Next, the partial derivative r of the target laser stripe image is calculated. x r y r xx r yy and r xy The direction with the largest absolute value of the second derivative is the direction of the laser stripe normal at the current position; this direction is determined by calculating the eigenvalues ​​and eigenvectors of the Hessian matrix, which is:

[0070]

[0071] Then we obtain the eigenvector (n) corresponding to the largest eigenvalue of the Hessian matrix. x ,n y );

[0072] Finally, the light intensity distribution function is constructed along the normal direction using Taylor expansion, and then the sub-pixel position of the center line is calculated:

[0073]

[0074] (p x ,p y )=(x0+tn x ,y0+tn y )

[0075] Among them, (p x ,p y Let (x0, y0) be the sub-pixel coordinates calculated at (x0, y0); since the Steger algorithm uses Taylor expansion, and Taylor expansion is only applicable in a small local range, it is only valid if the following conditions are met. The time coordinate calculation is valid.

[0076] like Figure 6 The image shows the result of extracting the center line of the laser stripe; the image shows the extraction result of the center line of the target laser stripe, while other laser stripes and interference were removed.

[0077] In another embodiment, a target laser stripe centerline extraction system based on visual simulation prior information is provided. The system, which applies the target laser stripe centerline extraction method of the above embodiment, includes: an image acquisition module, a visual simulation module, an image processing module, a filtering module, and a laser stripe centerline extraction module.

[0078] The image acquisition module is used to acquire images of the scene under test and obtain laser stripe images;

[0079] The visual simulation module is used to perform visual simulation of the scene under test and obtain the simulated laser stripe of the target laser stripe in the current state.

[0080] The image processing module is used to determine the region of interest based on the distribution range of the simulated laser stripes in the camera-acquired image, and to crop the camera-acquired image based on the region of interest to obtain the cropped laser stripe image.

[0081] The filtering module calculates the direction parameters based on the direction information of the obtained simulated laser stripes, establishes a two-dimensional Gabor filter, and uses the two-dimensional Gabor filter to filter the cropped laser stripe image, detects laser stripes with specific directions in the image, and obtains the target laser stripe.

[0082] The laser stripe centerline extraction module is used to extract the laser stripe centerline from the obtained target laser stripe image using the Steger algorithm.

[0083] In another embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the target laser stripe centerline extraction method as described in the above embodiment.

[0084] In another embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the target laser stripe centerline extraction method of the above embodiments.

[0085] It should also be noted that, in this specification, terms such as "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0086] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting the center line of a target laser stripe based on prior information from visual simulation, characterized in that, Includes the following steps: S1. Use a line structured light sensor to collect data on the scene under test to obtain a laser stripe image; S2. Perform visual simulation on the scene to be tested to obtain the simulated laser stripes of the target laser stripes in the current state; the visual simulation of the scene to be tested specifically involves: The imaging process of the vision sensor is reproduced in the digital twin system to obtain the acquisition results in a simulated environment. For the line structured light sensor, this means generating a simulated laser stripe image. S3. Determine the region of interest based on the distribution range of the simulated laser stripes in the camera-acquired image. Crop the camera-acquired image based on the region of interest to obtain the cropped laser stripe image, specifically: The region of interest (ROI) is determined based on the distribution of simulated laser stripes in the camera-acquired image, so that the target stripes are located within the ROI for further processing. Considering the deviation between the actual situation and the simulation, the actual stripes are not completely consistent with the simulated stripes. Therefore, the range of the ROI is expanded to increase the adaptability of the algorithm. At the same time, the image size is taken into account to ensure that the ROI does not exceed the range of the camera-acquired image. The specific formula for calculating the region of interest is as follows: in, and These are the pixel coordinates of the top-left and bottom-right points of the rectangular region of interest, respectively. and These are simulated laser stripe dots in the camera-acquired image. , The pixel coordinates of the direction, and according to Sort the coordinates from smallest to largest; and These are the width and height of the image captured by the camera, respectively; S4. Calculate the direction parameters based on the obtained direction information of the simulated laser stripes, and establish a two-dimensional Gabor filter; the specific formula for calculating the direction parameters is as follows: in, To simulate the number of laser stripe dots; S5. Filter the cropped laser stripe image using a two-dimensional Gabor filter to detect laser stripes with specific directions in the image and obtain the target laser stripe. S6. Use the Steger algorithm to extract the center line of the laser stripe from the obtained target laser stripe image to obtain the center line of the target laser stripe.

2. The method for extracting the center line of a target laser stripe based on visual simulation prior information according to claim 1, characterized in that, Step S2 is as follows: First, the line structured light sensor is calibrated to obtain the camera's intrinsic and extrinsic parameters and the laser plane equation. Then, a laser plane is established in the simulation environment. Within the laser plane, a set of rays uniformly distributed at certain angular intervals are established with the laser emission origin as the center. The laser plane is discretized into several rays, and the intersection point of each ray with the model is calculated. The intersection line obtained by connecting the intersection points is the laser stripe to be obtained. The number of rays n determines the shape accuracy of the generated simulated laser stripe. The more rays there are, the higher the shape accuracy of the laser stripe. For each laser beam Find the intersection points of the ray and the workpiece model, and retain the intersection point closest to the origin of the laser emission. ; Connecting the intersections of all the rays yields the laser stripes formed on the surface of the virtual workpiece by the virtual line structured light sensor.

3. The method for extracting the center line of a target laser stripe based on visual simulation prior information according to claim 1, characterized in that, In step S4, the two-dimensional Gabor filter is specifically represented as follows: 。 4. The method for extracting the center line of a target laser stripe based on visual simulation prior information according to claim 3, characterized in that, In step S5, the formula for the real part of the two-dimensional Gabor filter is: in, These are the pixel coordinates in the laser stripe image; The wavelength of the filter kernel directly affects its size. The direction of the filter; This represents the phase shift of the tuning function; The standard deviation of the Gaussian kernel function; The aspect ratio of the kernel function determines the shape of the filter; The cropped laser stripe image is filtered using a two-dimensional Gabor filter and then segmented by thresholding to obtain laser stripes with a specific direction.

5. The method for extracting the center line of a target laser stripe based on visual simulation prior information according to claim 1, characterized in that, Step S6 is as follows: The Steger algorithm first applies a Gaussian filter to the target laser stripe image, the scale of which is... The following conditions must be met: in, It is the width of the laser stripe; Next, the partial derivatives of the target laser stripe image are calculated. , , , and The direction with the largest absolute value of the second derivative is the direction of the laser stripe normal at the current position; this direction is determined by calculating the eigenvalues ​​and eigenvectors of the Hessian matrix, which is: This leads to the eigenvector corresponding to the largest eigenvalue of the Hessian matrix. , ); Finally, the light intensity distribution function is constructed along the normal direction using Taylor expansion, and then the sub-pixel position of the center line is calculated: in, for The sub-pixel coordinates are calculated at [location]; since Taylor expansion is only applicable within a small local range, it is only valid if [condition] is satisfied. The time coordinate calculation is valid.

6. A target laser stripe centerline extraction system based on visual simulation prior information, characterized in that, The system comprising the target laser stripe centerline extraction method according to any one of claims 1-5 includes: Image acquisition module, visual simulation module, image processing module, filtering module, and laser stripe center line extraction module; The image acquisition module is used to acquire images of the scene under test and obtain laser stripe images; The visual simulation module is used to perform visual simulation of the scene under test and obtain the simulated laser stripe of the target laser stripe in the current state. The image processing module is used to determine the region of interest based on the distribution range of the simulated laser stripes in the camera-acquired image, and to crop the camera-acquired image based on the region of interest to obtain the cropped laser stripe image. The filtering module calculates the direction parameters based on the direction information of the obtained simulated laser stripes, establishes a two-dimensional Gabor filter, and uses the two-dimensional Gabor filter to filter the cropped laser stripe image, detects laser stripes with specific directions in the image, and obtains the target laser stripe. The laser stripe centerline extraction module is used to extract the laser stripe centerline from the obtained target laser stripe image using the Steger algorithm.

7. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes a computer program, it implements the target laser stripe centerline extraction method according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the target laser stripe centerline extraction method according to any one of claims 1-5.

Images