A line structured light processing method and system based on FPGA
Through the parallel processing of FPGA and image block segmentation technology, high efficiency of linear structured light processing is achieved, the problem of low efficiency in the prior art is solved, and the processing time is shortened to microsecond level.
Patent Information
- Application Number
- CN202410954557.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-17
AI Technical Summary
The prior art has low efficiency in online structured light processing, especially in dealing with the effects of halo and stray light, and the processing speed is not sufficient to meet real-time requirements.
Parallel processing is performed through FPGA, grayscale pixel data in parallel format is obtained, image channel layer and image block are divided, IP is extracted using several light bars for parallel processing, generate sub-pixel coordinates of the laser bar center, and perform ping-pong operations through BRAM to cache images block, realizes fast optical bar center coordinate extraction.
The linear structure light processing time is shortened to the microsecond level, which improves the processing efficiency and solves the problem of low efficiency in the prior art.
Smart Images

Figure CN118505730B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a line structured light processing method and system based on FPGA. Background Art
[0002] In the field of active vision measurement, structured light measurement is a common measurement method, in which line laser features are widely used. Line lasers are used to project laser strips onto the object to be measured, and the light strip images are obtained through image acquisition equipment. In the image analysis process, extracting the center of the light strip is a crucial step. However, current methods face some challenges in accurately and quickly extracting the light strips in each frame of the image and dealing with the effects of halo and stray light. In order to meet these challenges, in recent years, researchers have begun to focus on using hardware such as FPGA to improve the efficiency of light strip center extraction. However, most of the related studies use FPGA for simple acceleration processing, such as using grayscale centroid or Steger method. These studies have certain limitations in dealing with the effects of halo and stray light, and their processing speed is at the ms level, which still needs to be further improved.
[0003] For example, patent application No. 201811414940X discloses a fast processing system for line structured light images based on SOC-FPGA. Specifically, the calculation method in the image processing algorithm IP core in the FPGA can realize the extraction of the center coordinates of the laser bar in parallel. It can be seen that this solution only uses the IP core in the FPGA to extract the center coordinates of the laser bar in parallel, which generally takes about 40ms to process a frame of video image, which is inefficient.
[0004] Currently, no effective solution has been proposed for the problem of how to improve the efficiency of line structured light processing in related technologies. Summary of the invention
[0005] The embodiments of the present application provide a line structured light processing method and system based on FPGA to at least solve the problem of how to improve the efficiency of line structured light processing in the related art.
[0006] In a first aspect, an embodiment of the present application provides a line structured light processing method based on FPGA, the method comprising:
[0007] Acquire grayscale pixel data in a parallel format, wherein the grayscale pixel data is data obtained based on line structured light processing;
[0008] Segmenting a single frame of video image in the grayscale pixel data to obtain a first number of image channel layers; segmenting the image channel layers again to obtain a second number of image blocks;
[0009] Extract IP through a plurality of light strips, process the image blocks of the image channel layer in parallel, and obtain the sub-pixel coordinates of the centers of a plurality of laser strips;
[0010] The plurality of laser bar center sub-pixel coordinates are spliced to obtain a data sequence containing complete laser bar center sub-pixel coordinates.
[0011] In some embodiments, before extracting IP through a plurality of light strips and processing the image blocks of the image channel layer in parallel to obtain the sub-pixel coordinates of the centers of a plurality of laser strips, the method includes:
[0012] Based on the image block reading cycle and the image block writing cycle, a ping-pong operation is performed through a block random access memory BRAM in a programmable logic gate array FPGA to cache and read the image blocks of the image channel layer.
[0013] In some embodiments, re-segmenting the image channel layer to obtain a second number of image blocks includes:
[0014] The image channel layer is segmented based on the characteristics of the optical sensor CMOS to obtain an upper half image channel layer and a lower half image channel layer;
[0015] Re-segmenting the upper image channel layer to obtain a second number of image blocks of the upper image channel layer;
[0016] The lower half image channel layer is segmented again to obtain a second number of image blocks of the lower half image channel layer.
[0017] In some embodiments, the IP is extracted by a plurality of light strips, and the image blocks of the image channel layer are processed in parallel to obtain the sub-pixel coordinates of the centers of a plurality of laser strips, including:
[0018] Extract IP through a plurality of light strips, process the image blocks of the upper half image channel layer in parallel, and obtain the sub-pixel coordinates of the centers of a plurality of laser strips;
[0019] The IP is extracted through a plurality of light strips, and the image blocks of the lower half image channel layer are processed in parallel to obtain the sub-pixel coordinates of the centers of a plurality of laser strips.
[0020] In some embodiments, the IP is extracted by a plurality of light strips, and the image blocks of the upper half image channel layer are processed in parallel to obtain the sub-pixel coordinates of the centers of a plurality of laser strips, including:
[0021] Generate local thresholds based on the grayscale features of the second number of image blocks in the upper image channel layer respectively;
[0022] Filtering the corresponding image block to filter out pixels in the image block that are smaller than the local threshold;
[0023] Extracting the best connected domain of each column in the filtered image block;
[0024] Based on the optimal connected domain, several sub-pixel coordinates of the center of the laser bar are calculated, wherein the sub-pixel coordinates of the center of the laser bar include a height coordinate and a width coordinate, the height coordinate is the grayscale centroid of the optimal connected domain, and the width coordinate is the serial number of the optimal connected domain.
[0025] In some embodiments, the IP is extracted by a plurality of light strips, and the image blocks of the lower half image channel layer are processed in parallel to obtain a plurality of laser strip center sub-pixel coordinates including:
[0026] Generate local thresholds based on the grayscale features of the second number of image blocks in the lower half image channel layer respectively;
[0027] Filtering the corresponding image block to filter out pixels in the image block that are smaller than the local threshold;
[0028] Extracting the best connected domain of each column in the filtered image block;
[0029] Based on the optimal connected domain, several sub-pixel coordinates of the center of the laser bar are calculated, wherein the sub-pixel coordinates of the center of the laser bar include a height coordinate and a width coordinate, the height coordinate is the grayscale centroid of the optimal connected domain, and the width coordinate is the serial number of the optimal connected domain.
[0030] In some embodiments, segmenting a single frame of video image in the grayscale pixel data to obtain a first number of image channel layers includes:
[0031] The single-frame video image in the grayscale pixel data is segmented to obtain 32 image channel layers, wherein the size of the single-frame video image is 1216 pixels*2048 pixels, and the size of the image channel layer is 1216 pixels*64 pixels.
[0032] In some embodiments, re-segmenting the image channel layer to obtain a second number of image blocks includes:
[0033] The image channel layer is divided again to obtain 38 image blocks, wherein the size of the image block is 16 pixels*64 pixels.
[0034] In some embodiments, obtaining grayscale pixel data in a parallel format includes:
[0035] Obtaining grayscale pixel data in serial format through low voltage differential signal LVDS;
[0036] The grayscale pixel data is deserialized to obtain grayscale pixel data in a parallel format.
[0037] In a second aspect, an embodiment of the present application provides a line structured light processing system based on FPGA, the system is used to execute the method described in the first aspect above, the system includes a laser, an optical sensor CMOS and a programmable logic gate array FPGA;
[0038] The laser is used to emit line laser to perform line structured light processing;
[0039] The optical sensor CMOS is used to generate grayscale pixel data under line structured light processing, and send the grayscale pixel data to the programmable logic gate array FPGA through a low voltage differential signal LVDS;
[0040] The programmable logic gate array FPGA is used to control the intensity of the line laser emitted by the laser through a PWM duty cycle;
[0041] The programmable logic gate array FPGA is used to obtain grayscale pixel data in a parallel format, wherein the grayscale pixel data is data obtained based on line structured light processing; segment a single frame video image in the grayscale pixel data to obtain a first number of image channel layers; segment the image channel layer again to obtain a second number of image blocks; extract IPs through a plurality of light strips, and process the image blocks of the image channel layer in parallel to obtain a plurality of laser strip center sub-pixel coordinates; and splice the plurality of laser strip center sub-pixel coordinates to obtain a data sequence containing a complete laser strip center sub-pixel coordinate.
[0042] Compared with the related art, the embodiment of the present application provides a line structured light processing method and system based on FPGA, wherein the method obtains grayscale pixel data in a parallel format, wherein the grayscale pixel data is data obtained based on line structured light processing; segments a single-frame video image in the grayscale pixel data to obtain a first number of image channel layers; segments the image channel layer again to obtain a second number of image blocks; extracts IPs through a number of light bars, and processes the image blocks of the image channel layer in parallel to obtain a number of laser bar center sub-pixel coordinates; splices the sub-pixel coordinates of the laser bar centers to obtain a data sequence containing complete laser bar center sub-pixel coordinates, thereby realizing segmentation of the image channel layer of a single-frame image into small-sized image blocks, and only caching smaller-sized image blocks for parallel calculation of the light bar center coordinates, without waiting for the calculation results of the coordinates, and only blocking the write cycle of one image block to process the next frame of the image, thereby shortening the processing time of a frame of the image in the structured light measurement to microseconds, and solving the problem of how to improve the efficiency of line structured light processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0044] Figure 1 is a flowchart of the steps of a line structured light processing method based on FPGA according to an embodiment of the present application;
[0045] Figure 2 is a schematic diagram of dividing a single-frame video image into image channels and image blocks according to an embodiment of the present application;
[0046] Figure 3 is a timing diagram of accessing an image block through a ping-pong operation through a BRAM according to an embodiment of the present application;
[0047] Figure 4 is a flow chart of extracting IP from light strips for parallel processing according to an embodiment of the present application;
[0048] Figure 5 is a schematic diagram of the effect of smoothing filtering according to an embodiment of the present application;
[0049] Figure 6 is a schematic diagram of a process for extracting an optimal connected domain according to an embodiment of the present application;
[0050] Figure 7 is a characteristic schematic diagram of a halo and a light strip according to an embodiment of the present application;
[0051] Figure 8 is a schematic structural diagram of a line structured light processing system based on FPGA according to an embodiment of the present application;
[0052] Fig. 9 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application.
[0053] Figure symbols: 300, single-frame video image; 310, lower-half image channel layer; 311, upper-half image channel layer; 320, image block; 401, light stripes before image 2D filtering; 402, light stripes after image 2D filtering. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0055] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.
[0056] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0057] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.
[0058] The present application embodiment provides a line structured light processing method based on FPGA. Figure 1 is a flowchart of the steps of the line structured light processing method based on FPGA according to an embodiment of the present application, such as Figure 1As shown, the method comprises the following steps:
[0059] Step S102, obtaining grayscale pixel data in a parallel format, wherein the grayscale pixel data is data obtained based on line structured light processing;
[0060] Specifically, step S102 obtains grayscale pixel data in a serial format through a low voltage differential signal LVDS; deserializes the grayscale pixel data to obtain grayscale pixel data in a parallel format.
[0061] It should be noted that low-voltage differential signal LVDS (Low-Voltage Differential Signaling) is an electrical signal transmission standard. It uses a very low voltage swing (approximately between 250mV and 450mV) to transmit data at high speed through a pair of balanced wires (usually PCB traces or dedicated balanced cables). In order to increase the data transmission rate and reduce the number of physical lines, the data is serialized and sent through the LVDS link. The receiving end needs to use a deserializer to restore these high-speed serial LVDS signals back to their original parallel data form for subsequent processing or storage. In other words, converting the data stream (grayscale pixel data) of multiple LVDS channels from a serial format back to a parallel format in step S102 can increase the data transmission rate and reduce the number of physical lines, thereby improving the efficiency of line structured light processing.
[0062] Step S104, segmenting the single-frame video image in the grayscale pixel data to obtain a first number of image channel layers; segmenting the image channel layers again to obtain a second number of image blocks;
[0063] Step S104 specifically includes the following steps:
[0064] Step S1041, segmenting the single-frame video image in the grayscale pixel data to obtain 32 image channel layers, wherein the size of the single-frame video image is 1216 pixels * 2048 pixels, and the size of the image channel layer is 1216 pixels * 64 pixels. Furthermore, the size of the upper half of each image layer is 608 pixels * 64 pixels, and the size of the lower half is also 608 pixels * 64 pixels.
[0065] It should be noted that the image channel layers are not limited to 32. A single-frame video image can also be divided into any number of image channel layers, such as 1 image channel layer. In this way, although the processing time of a single-frame video image will increase from 6us (microseconds) to 190us (microseconds), the hardware cost can be reduced.
[0066] Step S1042, the image channel layer is divided again to obtain 38 image blocks, wherein the size of the image block is 16 pixels*64 pixels.
[0067] Specifically, step S1042: Figure 2 is a schematic diagram of dividing a single frame video image into image channels and image blocks according to an embodiment of the present application, such as Figure 2 As shown, based on the characteristics of the optical sensor CMOS, the image channel layer is segmented to obtain an upper image channel layer and a lower image channel layer; the upper image channel layer is segmented again to obtain a second number of image blocks of the upper image channel layer; the lower image channel layer is segmented again to obtain a second number of image blocks of the lower image channel layer.
[0068] It should be noted that the division into the upper image channel layer and the lower image channel layer in the embodiment is to adapt to the pixel timing output by the optical sensor CMOS and avoid using memory to cache the image frame.
[0069] Step S106, extracting IPs through a plurality of light strips, performing parallel processing on image blocks of the image channel layer, and obtaining sub-pixel coordinates of the centers of a plurality of laser strips;
[0070] Before step S106, the method further includes step S105, based on the image block reading cycle and the image block writing cycle, performing ping-pong operation through the block random access memory BRAM in the programmable logic gate array FPGA to cache and read the image blocks of the image channel layer.
[0071] Step S105 is preferably: Figure 3 is a timing diagram of accessing an image block through a ping-pong operation through a BRAM according to an embodiment of the present application, such as Figure 3 As shown, it can be seen that the theoretical blocking cycle for processing a frame of image in this embodiment is one "image block write cycle". Increasing the number of pixel rows of the image block will increase the processing cycle, and at the same time will increase the consumption of BRAM resources inside the FPGA. However, too small a number of rows is also inappropriate, because each time a column of pixels is read or processed, two pixel rows of idle time are required for scheduling before the next column of pixels can be processed. Therefore, the size of the image block is 16 pixels * 64 pixels. In this embodiment, the total number of "image block read cycles" is 1152, and its calculation formula 1152=(16+2)*64, where 16 is the number of pixel rows of the image block, 64 is the number of pixel columns of the image block, and 2 is the idle time required for reading; the calculation formula of the "image block write cycle" is 1152=(64+8)*16, where 16 is the number of pixel rows of the image block, 64 is the number of pixel columns of the image block, and 8 is the idle time required for writing.
[0072] It should be noted that BRAM is shared for multiple image channel layers, because the timing between multiple image channel layers of the same frame of video image is completely synchronized, only the pixel grayscale is different, so the address line of each image channel layer to BRAM is completely equal in the same cycle, so the data of all image channels can be spliced together and the address lines are shared. Since the data bit width of the smallest unit of BRAM in FPGA is 36 / 18, and the pixel grayscale bit width is only 8 bits, if one image channel occupies one BRAM, the remaining 28 / 10 data bit width in BRAM is wasted. Therefore, the method in this embodiment can save BRAM resources inside FPGA and reduce the final hardware cost.
[0073] In addition, in some other embodiments, a local threshold is generated while caching an image block, and then the local threshold is applied when reading the image block; based on this idea, the grayscale mean of each row of the image block can also be cached to optimize the subsequent connected domain extraction. Because the grayscale in the horizontal direction around the center of the light strip is likely to be larger, while the grayscale in the horizontal direction around the stray light or noise is smaller, this is determined by the characteristics of the line laser. Because the light strip generated by the line laser is parallel to the horizontal axis of the camera, the captured image will not have completely vertical light strips, only horizontal light strips or slightly inclined light strip segments. In the embodiment, an additional BRAM is instantiated specifically for caching horizontal grayscale means. The address depth of this BRAM is 32, and the 16-row grayscale mean of an image block is also written and read using ping-pong operations.
[0074] Step S106 specifically includes the following steps:
[0075] Step S1061, extracting IPs through a plurality of light strips, performing parallel processing on the image blocks of the upper half image channel layer, and obtaining a plurality of sub-pixel coordinates of the centers of the laser strips;
[0076] Figure 4 FIG. 1 is a flow chart of extracting IP from light strips for parallel processing according to an embodiment of the present application. Figure 4 As shown, step S1061 specifically includes the following steps:
[0077] Step S611, generating local thresholds based on the grayscale features of a second number of image blocks in the upper image channel layer;
[0078] It should be noted that the local threshold is generated based on the grayscale features of the image block. This is because the grayscale distribution of the light strips on the entire image is not uniform. If a fixed threshold is used to perform threshold filtering on the entire image, the light strips in the darker area will be incorrectly filtered out, and the denoising effect in the brighter area is not so good. Therefore, the embodiment of the present application divides the upper half of the image channel layer into image blocks, and then generates a local threshold for filtering, which can achieve a better filtering effect. The calculation formula for the local threshold is: local threshold = minimum grayscale of the image block + maximum grayscale of the image block / 8. In addition, the local threshold can also be generated based on the histogram features of the image block to reduce the amount of calculation.
[0079] Step S612, filtering the corresponding image block to filter out pixels in the image block that are smaller than a local threshold;
[0080] It should be noted that the filtering operation can use smoothing filtering, which can effectively reduce the thermal noise and high-frequency noise generated in the optical sensor CMOS shooting and improve the continuity of the final generated light strip. Figure 5 It is a schematic diagram of the effect of smoothing filtering according to an embodiment of the present application. In this embodiment, 9 pixel grayscales (3x3 operator) are preferably used for smoothing filtering, and pixels below the threshold are filtered (the grayscale is directly set to 0). In subsequent calculations, pixels with grayscale 0 can be used to segment each connected domain, where the grayscale value ranges from 0 to 255, grayscale 0 represents black, and grayscale 255 represents 100% white.
[0081] Step S613, extracting the best connected domain of each column in the filtered image block;
[0082] It should be noted that Figure 6 is a schematic diagram of a process for extracting the best connected domain according to an embodiment of the present application, such as Figure 6 As shown, step S613 is responsible for extracting the one-dimensional connected domain of each pixel column in the image block. After all image blocks are traversed, the upper image channel layer will output: the best connected domain of 64 columns, the extended connected domain of 64 columns; the lower image channel and the upper image channel simultaneously output: the best connected domain of 64 columns, the extended connected domain of 64 columns. Since the passband of the upper image channel layer extends from the top to the middle, and the lower image channel layer extends from the bottom to the middle, a connected domain may appear across the upper and lower image channel layers. At this time, it is necessary to splice the extended connected domain of the upper 64 columns and the extended connected domain of the lower 64 columns to generate a spliced connected domain (connected domain A). At this time, three connected domains are obtained (connected domain A, the best connected domain of the upper half, and the best connected domain of the lower half). One of the three connected domains is selected as the best connected domain of the image channel for final output.
[0083] It should be further explained that the connected domain calculations in the embodiment are all split into pipelines, which will not increase the processing time between two frames of video images. When the current video image is being collected, the previous frame of the video image is processed at the same time, and the two do not conflict. This embodiment innovatively splits the 64 best connected domain calculation tasks of a single image channel into multiple levels of pipelines, reducing FPGA delay and improving throughput.
[0084] The structure of the connected domain is as follows:
[0085] Connected domain structure: {
[0086] Cnt / / Connected domain pixel width
[0087] GrayAccum / / connected domain grayscale sum
[0088] Gray2DAccum / / The mean value of the horizontal grayscale of the connected domain
[0089] WeightAccum / / The sum of the connected domain [grayscale * coordinate]
[0090] Gradient / / Current gradient of connected domain
[0091] GrayMax / / Maximum grayscale of connected domain
[0092] Scores / / Connectivity domain selection weight
[0093] }
[0094] Among them, Scores (connected domain selection weight) calculation takes a lot of time and needs to wait for other data to be updated before calculation. The connected domain weight calculation formula Scores = (Gray2DAccum+GrayAccum) / Cnt, the division calculation in the formula is completed by the divider IP, Gray2DAccum is the accumulation result of the horizontal grayscale mean, which can improve the recognition rate; another solution is Scores = (GrayAccum-GrayMax) / Cnt, this solution method can reduce the weight of the narrower bright connected domain, which is suitable for high exposure scenes.
[0095] The calculation of connected domains is mainly to accumulate. For example, the grayscale sum of the connected domain width in the connected domain structure is the result of accumulation. If there is a connected domain that spans multiple image blocks, the connected domain result being accumulated needs to be saved. When the next image block comes, the accumulated result of the previous image block is loaded and the accumulation continues. This process is called "updating and extending the connected domain". The segmentation of the connected domain relies on the gradient. The gradient of the pixel with a grayscale of 0 is 0. If the current row pixel is greater than the previous row pixel value, the gradient is 1. If the current row pixel is less than the previous row pixel, the gradient is -1. If the current row pixel is equal to the previous row pixel, the gradient remains unchanged. When updating and extending the connected domain, as long as the gradient is not equal to 0, it will be expanded. The data in the connected domain is also accumulated or updated normally. When encountering a pixel with a gradient of 0, it means that a connected domain statistics is completed. The pixel can be called the "end point". In addition, if the current row gradient is 1 and the previous row is -1, the pixel can be called a "saddle point". At this time, it is both the beginning of a connected domain and the end of the previous connected domain. Saving and loading of connected domains do not need to wait for each other and can be performed simultaneously.
[0096] There is a slight difference between the update of the best connected domain and the update of the extended connected domain. The extended connected domain is updated at pixels with non-zero gradients, while the best connected domain is updated at the "end point" and "saddle point". The selection weights of the best connected domain and the extended connected domain are compared at the "end point" and "saddle point". If the selection weight of the extended connected domain is greater than the selection weight of the current best connected domain, then the best connected domain copies the content of the extended connected domain, and the extended connected domain will be reset immediately at the next moment. If it is the "end point", all will be reset to 0. After all image blocks are counted, 64 extended connected domains and 64 best connected domains are obtained. These data begin to be output in ascending order of columns at the same time. Correspondingly, the 64 extended connected domains and 64 best connected domains of the other half of the image channels are also output at the same time.
[0097] Next, you need to splice and compare the four connected domains in each column and output a connected domain. This process is also divided into pipelines. The extended connected domains of the upper and lower halves are spliced, and the connected domain generated by the splicing is called connected domain A. The splicing calculation only requires the accumulation of the connected domain width, grayscale sum, horizontal grayscale mean sum, and [grayscale * vertical coordinate] sum. The connected domain gradient has no effect at this time. The maximum grayscale of the connected domain is assigned the maximum value of the maximum grayscale of the two extended connected domains, and the connected domain selection weight is recalculated. At the same time, the best connected domains of the upper image channel layer and the lower image channel layer are compared, and the connected domain with a larger weight is selected, which is called connected domain B. Then, connected domain A and connected domain B are compared, and the connected domain with a larger weight is selected to generate connected domain C. Connected domain C is the final best connected domain of an image channel, such as Figure 6 shown.
[0098] Step S614, based on the best connected domain, calculate and obtain several sub-pixel coordinates of the center of the laser bar, wherein the sub-pixel coordinates of the center of the laser bar include a height coordinate and a width coordinate, the height coordinate is the grayscale centroid of the best connected domain, and the width coordinate is the serial number of the best connected domain.
[0099] It should be noted that the grayscale centroid within the best connected domain is calculated as the height coordinate of the light strip center, and the column number is used as the width coordinate of the light strip center. The traditional grayscale centroid method is more susceptible to interference from noise or halo, but the connected domain method in this embodiment filters out halo and stray light, and then uses the grayscale centroid method to generate the sub-pixel light strip center coordinate. Figure 7 It is a characteristic schematic diagram of the halo and light strip according to an embodiment of the present application.
[0100] Step S1062, extracting IP through a plurality of light strips, performing parallel processing on the image blocks of the lower half image channel layer, and obtaining a plurality of sub-pixel coordinates of the centers of the laser strips.
[0101] Step S621, generating local thresholds based on the grayscale features of a second number of image blocks in the lower half image channel layer;
[0102] Step S622, filtering the corresponding image block to filter out pixels in the image block that are smaller than a local threshold;
[0103] Step S623, extracting the best connected domain of each column in the filtered image block;
[0104] Step S624, based on the best connected domain, calculate and obtain several sub-pixel coordinates of the center of the laser bar, wherein the sub-pixel coordinates of the center of the laser bar include a height coordinate and a width coordinate, the height coordinate is the grayscale centroid of the best connected domain, and the width coordinate is the serial number of the best connected domain.
[0105] It should be noted that the specific operations of step S621 to step S624 are the same as those of the above-mentioned step S611 to step S614.
[0106] Step S108, splicing the sub-pixel coordinates of the centers of several laser bars to obtain a data sequence containing the complete sub-pixel coordinates of the centers of the laser bars.
[0107] Steps S102 to S106 process 32 image channels in parallel, and each image channel outputs 64 light bar center sub-pixel coordinates; step S108 specifically splices 32x64 light bar center sub-pixel coordinates and expands them into a data sequence of 2048 length; converts the 2048-length light bar center sequence into a network data packet; sends the network data packet carrying the light bar sequence to a server (such as a personal computer) via Ethernet; the personal computer parses the network data packet through the host computer software to obtain a light bar sequence of 2048 length.
[0108] Through the above steps in the embodiment of the present application, the image channel layer of a single frame image is divided into small-sized image blocks, and only the smaller-sized image blocks are cached to perform parallel calculation of the center coordinates of the light strip. There is no need to wait for the calculation results of the coordinates. Only the write cycle of one image block is blocked to process the next frame of the image, which shortens the processing time of one frame of the image in the structured light measurement to microseconds, solving the problem of how to improve the efficiency of line structured light processing.
[0109] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0110] The embodiment of the present application provides a line structured light processing system based on FPGA. Figure 8 is a schematic diagram of the structure of a line structured light processing system based on FPGA according to an embodiment of the present application, such as Figure 8 As shown, the system includes a laser, an optical sensor CMOS and a programmable logic gate array FPGA;
[0111] A laser, used for emitting a line laser to perform line structured light processing;
[0112] The optical sensor CMOS is used to generate grayscale pixel data under line structured light processing and send the grayscale pixel data to the programmable logic gate array FPGA through a low voltage differential signal LVDS;
[0113] A programmable logic gate array FPGA is used to control the intensity of the line laser emitted by the laser through a PWM duty cycle;
[0114] A programmable logic gate array FPGA is used to obtain grayscale pixel data in a parallel format, wherein the grayscale pixel data is data obtained based on line structured light processing; segment a single frame video image in the grayscale pixel data to obtain a first number of image channel layers; segment the image channel layers again to obtain a second number of image blocks; extract IPs through a plurality of light strips, and process the image blocks of the image channel layers in parallel to obtain a plurality of laser strip center sub-pixel coordinates; and splice the plurality of laser strip center sub-pixel coordinates to obtain a data sequence containing a complete laser strip center sub-pixel coordinate.
[0115] Through the laser, optical sensor CMOS and programmable logic gate array FPGA in the embodiments of the present application, it is possible to divide the image channel layer of a single frame image into small-sized image blocks, and only cache the smaller-sized image blocks to perform parallel calculation of the center coordinates of the light strip. There is no need to wait for the calculation results of the coordinates. Only the write cycle of one image block is blocked to process the next frame of the image, which shortens the processing time of one frame of the image in structured light measurement to microseconds, solving the problem of how to improve the efficiency of line structured light processing.
[0116] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0117] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0118] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0119] In addition, in combination with the FPGA-based line structured light processing method in the above embodiments, the present application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the FPGA-based line structured light processing methods in the above embodiments is implemented.
[0120] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a line structure light processing method based on FPGA is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.
[0121] In one embodiment, Fig. 9 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Fig. 9 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Fig. 9 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement a line structured light processing method based on FPGA, and the database is used to store data.
[0122] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0123] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0124] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0125] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A line structured light processing method based on FPGA, characterized in that: The method comprises: Acquire grayscale pixel data in a parallel format, wherein the grayscale pixel data is data obtained based on line structured light processing; Segmenting a single-frame video image in the grayscale pixel data to obtain a first number of image channel layers; segmenting the image channel layer based on the characteristics of the optical sensor CMOS to obtain an upper half image channel layer and a lower half image channel layer; segmenting the upper half image channel layer again to obtain a second number of image blocks of the upper half image channel layer; segmenting the lower half image channel layer again to obtain a second number of image blocks of the lower half image channel layer; Extracting IP through a plurality of light strips, processing the image blocks of the upper half image channel layer in parallel, and obtaining a plurality of sub-pixel coordinates of the center of the laser strips; extracting IP through a plurality of light strips, processing the image blocks of the lower half image channel layer in parallel, and obtaining a plurality of sub-pixel coordinates of the center of the laser strips; The plurality of laser bar center sub-pixel coordinates are spliced to obtain a data sequence containing complete laser bar center sub-pixel coordinates.
2. The method according to claim 1, characterized in that Before extracting IP through a plurality of light strips and processing the image blocks of the image channel layer in parallel to obtain the sub-pixel coordinates of the centers of a plurality of laser strips, the method comprises: Based on the image block reading cycle and the image block writing cycle, a ping-pong operation is performed through a block random access memory BRAM in a programmable logic gate array FPGA to cache and read the image blocks of the image channel layer.
3. The method according to claim 1, characterized in that The IP is extracted by a plurality of light strips, and the image blocks of the upper half image channel layer are processed in parallel to obtain the sub-pixel coordinates of the centers of a plurality of laser strips, including: Generate local thresholds based on the grayscale features of the second number of image blocks in the upper image channel layer respectively; Filtering the corresponding image block to filter out pixels in the image block that are smaller than the local threshold; Extracting the best connected domain of each column in the filtered image block; Based on the optimal connected domain, several sub-pixel coordinates of the center of the laser bar are calculated, wherein the sub-pixel coordinates of the center of the laser bar include a height coordinate and a width coordinate, the height coordinate is the grayscale centroid of the optimal connected domain, and the width coordinate is the serial number of the optimal connected domain.
4. The method according to claim 1, characterized in that The IP is extracted through several light strips, and the image blocks of the lower half image channel layer are processed in parallel to obtain several sub-pixel coordinates of the center of the laser strips, including: Generate local thresholds based on the grayscale features of the second number of image blocks in the lower half image channel layer respectively; Filtering the corresponding image block to filter out pixels in the image block that are smaller than the local threshold; Extracting the best connected domain of each column in the filtered image block; Based on the optimal connected domain, several sub-pixel coordinates of the center of the laser bar are calculated, wherein the sub-pixel coordinates of the center of the laser bar include a height coordinate and a width coordinate, the height coordinate is the grayscale centroid of the optimal connected domain, and the width coordinate is the serial number of the optimal connected domain.
5. The method according to claim 1, characterized in that Segmenting a single frame of video image in the grayscale pixel data to obtain a first number of image channel layers includes: The single-frame video image in the grayscale pixel data is segmented to obtain 32 image channel layers, wherein the size of the single-frame video image is 1216 pixels*2048 pixels, and the size of the image channel layer is 1216 pixels*64 pixels.
6. The method according to claim 5, characterized in that The image channel layer is segmented again to obtain a second number of image blocks, including: The image channel layer is divided again to obtain 38 image blocks, wherein the size of the image block is 16 pixels*64 pixels.
7. The method according to claim 1, characterized in that Obtaining grayscale pixel data in parallel format involves: Obtaining grayscale pixel data in serial format through low voltage differential signal LVDS; The grayscale pixel data is deserialized to obtain grayscale pixel data in a parallel format.
8. A line structured light processing system based on FPGA, characterized in that: The system is used to perform the method described in any one of claims 1 to 7, and the system comprises a laser, an optical sensor CMOS and a programmable logic gate array FPGA; The laser is used to emit line laser to perform line structured light processing; The optical sensor CMOS is used to generate grayscale pixel data under line structured light processing, and send the grayscale pixel data to the programmable logic gate array FPGA through a low voltage differential signal LVDS; The programmable logic gate array FPGA is used to control the intensity of the line laser emitted by the laser through a PWM duty cycle; The programmable logic gate array FPGA is used to obtain grayscale pixel data in a parallel format, wherein the grayscale pixel data is data obtained based on line structured light processing; segment a single frame video image in the grayscale pixel data to obtain a first number of image channel layers; segment the image channel layer again to obtain a second number of image blocks; extract IPs through a plurality of light strips, and process the image blocks of the image channel layer in parallel to obtain a plurality of laser strip center sub-pixel coordinates; and splice the plurality of laser strip center sub-pixel coordinates to obtain a data sequence containing a complete laser strip center sub-pixel coordinate.
