Parallel filtering method and system of high-frame-rate CMOS image sensor

The CMOS image sensor is filtered in parallel through FPGA technology and ping-pong operation method, which solves the problem that the image processing speed cannot keep up in the traditional technology, and realizes efficient and real-time image processing, which is suitable for high-speed photography and machine vision.

CN118972712AInactive Publication Date: 2024-11-15HANGZHOU LINGXI ROBOT INTELLIGENT TECH CO LTD

Patent Information

Application Number
CN202411435322.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-15
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the prior art, the high frame rate data processing efficiency of CMOS image sensors is low, resulting in a mismatch between the processing speed and the front-end sensor shooting speed, limiting the overall system performance.

Method used

Using FPGA technology and ping-pong operation method, multi-channel image data is processed in parallel through multiple sliding filter windows, and weighted summing is performed through weighted matrix to achieve real-time and efficient image filtering processing.

Benefits of technology

It ensures that the efficiency of CMOS image data processing matches the shooting speed of the front-end sensor, improves the real-time, accuracy and flexibility of image processing, and provides more powerful technical support for high-speed photography and machine vision.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972712A_ABST
    Figure CN118972712A_ABST
Patent Text Reader

Abstract

The invention relates to a parallel filtering method of a high-frame-rate CMOS (Complementary Metal Oxide Semiconductor) image sensor, which realizes real-time and efficient processing of output data of the high-frame-rate CMOS image sensor by adopting optimization means such as an FPGA (Field Programmable Gate Array) technology and ping-pong operation, and can ensure that the processing efficiency of CMOS image data is matched with the shooting speed of a front-end sensor. Therefore, more powerful technical support is provided for the application fields of high-speed photography, machine vision and the like. According to the scheme, the problem that the speed of a traditional image processing system cannot follow up is solved, the real-time performance, accuracy and flexibility of image processing are ensured, and a brand new solution is provided for the fields of machine vision, industrial intelligent detection and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image data processing, and in particular to a parallel image filtering method, system, computer device and computer-readable storage medium for a high frame rate CMOS image sensor. Background Art

[0002] CMOS sensor, full name Complementary Metal-Oxide-Semiconductor, is a solid imaging sensor that converts light signals into electrical signals and outputs them in the form of images. Although the front-end capture capability of CMOS sensors has reached a high frame rate, the back-end processing system is often unable to process these high-speed incoming image data in real time, which leads to a mismatch between the processing speed and the shooting speed of the front-end sensor, thus limiting the overall system performance.

[0003] In related technologies, FPGA (field programmable gate array) is used for filtering. The high parallel processing capability of FPGA makes it an ideal choice for accelerating image processing. However, current FPGA-based filtering solutions mostly use a 3x3 sliding window pipeline processing method, which is more efficient when processing image data of a single channel, but is not suitable for multi-channel parallel transmission scenarios in high frame rate CMOS image sensors. Summary of the invention

[0004] The embodiments of the present application provide a parallel filtering method, system, computer device and computer-readable storage medium for a high frame rate CMOS image sensor, so as to at least solve the problem of poor data processing efficiency of CMOS image sensors in the related art.

[0005] In a first aspect, an embodiment of the present application provides a parallel filtering method for a high frame rate CMOS image sensor, which is implemented by a field programmable gate array, and the method includes: Collecting raw images of multiple channels output by CMOS image sensors; The pixel data of each channel is written and read through a ping-pong operation of a single dual-port access device, and in the process of reading, parallel pixel expansion is performed on the edge pixels of each channel; Through multiple sliding filter windows, the pixel data read from each channel are filtered in parallel according to a preset step size and a preset direction, and the filtered pixel data in each window are weighted and summed through a weighting matrix to obtain an image filtering result, wherein the weighting matrix is ​​dynamically formulated according to task requirements.

[0006] In some embodiments, any one of the dual-port memory devices is equally divided into a first port and a second port, and the first port and the second port implement pipeline writing and reading of pixel data in one channel through alternating ping-pong operations. During reading, edge pixels of each channel are repeatedly read to support edge pixel expansion of adjacent channels.

[0007] In some embodiments, writing and reading pixel data of each channel through a ping-pong operation of a single dual-port accessor includes: Acting in a first buffering cycle, buffering the input pixel data of any channel to the first port; In a second buffering cycle, the input pixel data is cached to the second port and the input pixel data in the first port is read out at the same time by performing port switching through the input data selection unit; In a third buffering cycle, the input data selection unit performs port switching to switch the input data back to the first port, and simultaneously switches the input pixel data in the second port; The steps of the first buffer cycle, the second buffer cycle and the third buffer cycle are repeatedly cycled until all pixel data of any channel are read out, wherein edge pixels of each channel are repeatedly read during reading to support edge pixel expansion of adjacent channels.

[0008] In some of the embodiments, the field programmable gate array collects the original image through N channel data, and the upper half image and the lower half image of the original image occupy the upper half channel and the lower half channel of the N channels respectively.

[0009] In some embodiments, during the reading process, performing parallel edge pixel expansion on edge pixels of each channel includes: For a first target channel in any non-image overall edge region: respectively obtaining first pixel data, second pixel data and third pixel data of left, right and bottom adjacent channels of an upper half channel of the first target channel; Fill the right pixel points of the first pixel data to the left edge of the first target channel, fill the left pixel points of the second pixel data to the right edge of the first target channel, fill the lower edge pixel points of the third pixel data to the lower edge of the first target channel, and perform zero filling on the upper edge of the target channel; For the second target channel located in the left edge area of ​​the entire image and the third target channel located in the right edge area, zero padding is performed on the left and right sides respectively; Zero pad the upper edge of each channel.

[0010] In some embodiments, the method further comprises: Read the pixel data of each channel according to a specific sequence, wherein the specific sequence includes: At the beginning of the clock cycle, the rightmost pixel is read; In the middle clock cycle, the pixel data of each channel is read sequentially from left to right; At the end of the clock cycle, the leftmost pixel is read; At the rising edge of the start clock cycle, the rightmost pixel data of each channel other than the third target channel is filled to the left edge of the right adjacent channel, At the rising edge of the end clock cycle, the leftmost pixel data of each channel other than the second target channel is filled to the rightmost side of the adjacent channel.

[0011] In some embodiments, the weighting matrix includes a Gaussian matrix and an asymmetric matrix. Among them, the parameters of the weighting matrix are dynamically adjusted according to the requirements of the filtering task, the filtered pixel data in each filtering window are weighted summed up respectively through the dynamically adjusted weighting matrix, and the weight of each pixel point in the weighted summation result is adjusted to obtain image filtering results suitable for different scenarios.

[0012] In a second aspect, an embodiment of the present application provides a parallel filtering system for a high frame rate CMOS image sensor, which is implemented by a field programmable gate array. The system includes: an acquisition module, an access module, and a filtering module, wherein: The acquisition module is used to acquire original images of multiple channels output by the CMOS image sensor; The access module is used to write and read the pixel data of each channel through a ping-pong operation of a single dual-port access device, and in the reading process, parallel pixel expansion is performed on the edge pixels of each channel; The filtering module is used to perform parallel filtering processing on the pixel data read from each channel according to a preset step size and a preset direction through multiple sliding filtering windows, and to perform weighted summation on the filtered pixel data in each window through a weighting matrix to obtain an image filtering result, wherein the weighting matrix is ​​dynamically formulated according to task requirements.

[0013] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0015] Compared with related technologies, the parallel filtering method for a high frame rate CMOS image sensor provided in the embodiment of the present application realizes real-time and efficient processing of the output data of the high frame rate CMOS image sensor by adopting optimization means such as FPGA technology and ping-pong operation, and can ensure that the processing efficiency of CMOS image data matches the shooting speed of the front-end sensor, thereby providing more powerful technical support for application fields such as high-speed photography and machine vision. The present application solves the problem that the traditional image processing system cannot keep up with the speed, ensures the real-time, accuracy and flexibility of image processing, and provides a new solution for the fields of machine vision and industrial intelligent detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of a parallel filtering method for a high frame rate CMOS image sensor according to an embodiment of the present application; Figure 2 is a schematic diagram of a mapping relationship between the number of channels and a complete image according to an embodiment of the present application; Figure 3 is a schematic diagram of performing a ping-pong operation on an original image in a dual-port access device according to an embodiment of the present application; Figure 4 It is a schematic diagram of an overall processing flow of performing multi-channel parallel filtering on an original image according to an embodiment of the present application; Figure 5 is a structural block diagram of a parallel filtering system for a high frame rate CMOS image sensor according to an embodiment of the present application; Figure 6 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0018] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0019] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0020] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0021] With the continuous advancement of technology, CMOS image sensors have been able to achieve high frame rate image capture, which provides strong technical support for applications such as high-speed photography and real-time monitoring. However, the large amount of data brought by the high frame rate puts higher requirements on the subsequent image processing system. Traditional image processing systems usually require frame processing, such as noise reduction, sharpening and other filtering operations, to improve image quality or extract specific information.

[0022] In existing technologies, although the front-end capture capability of CMOS sensors has reached a high frame rate, the back-end processing system is often unable to process these high-speed incoming image data in real time, which results in a mismatch between the processing speed and the shooting speed of the front-end sensor, thereby limiting the overall system performance.

[0023] To solve this problem, a solution was proposed to use FPGA (field programmable gate array) for filtering. The high parallel processing capability of FPGA makes it an ideal choice for accelerating image processing. However, the current FPGA-based filtering solutions mostly use a 3x3 sliding window pipeline processing method, which is more efficient when processing image data of a single channel, but is not suitable for multi-channel parallel transmission scenarios in high frame rate CMOS image sensors.

[0024] In order to improve data transmission efficiency, high frame rate CMOS image sensors usually divide the entire image into multiple channels for parallel transmission, thereby increasing the transmission bandwidth to achieve high frame rate. However, there is currently a lack of filtering technology on the market that can effectively process this multi-channel parallel transmission of image data. This requires the development of a filtering system that can process multi-channel image data in parallel while maintaining a high frame rate to meet the needs of high frame rate CMOS image sensors.

[0025] In view of this, the present application provides a parallel image filtering solution that can match the output of high frame rate CMOS image sensors to achieve real-time and efficient image processing. This not only helps to improve the performance of the overall system, but also ensures that the quality of image processing matches the shooting speed of the front-end sensor, thereby providing more powerful technical support for application fields such as high-speed photography and machine vision.

[0026] Figure 1 is a flow chart of a parallel filtering method for a high frame rate CMOS image sensor according to an embodiment of the present application, which is implemented by FPGA, such as Figure 1 As shown, the process includes the following steps: S101, collecting original images of multiple channels output by a CMOS image sensor; First of all, it should be noted that FPGA (Field-Programmable Gate Array) is a semi-custom integrated circuit that allows users to define the functions of its internal logic circuits through programming after manufacturing. FPGA consists of a large number of logic blocks (such as lookup tables, triggers, etc.) and programmable interconnect resources, which can be configured according to the user's design to achieve specific hardware functions.

[0027] The present application scheme uses FPGA as the core processing unit. Through FPGA, efficient parallel computing can be achieved, thereby greatly improving the image processing speed, thereby ensuring synchronization with the data output of the high frame rate CMOS image sensor.

[0028] In step S101, the multi-channel output capability of the CMOS image sensor is utilized to simultaneously collect raw image data from multiple channels. Specifically, the data stream from the CMOS image sensor is received through multiple parallel interfaces of the FPGA. In an exemplary embodiment, the FPGA receives image data of 64 channels, each channel has a size of 64 columns*608 rows, and the size of the complete image is 2048 columns*1216 rows.

[0029] Figure 2 is a schematic diagram of a mapping relationship between the number of channels and a complete image according to an embodiment of the present application, such as Figure 2 As shown, there are 64 channels in total, 32 channels for the upper half image and 32 channels for the lower half image. Generally, high frame rate CMOS image sensors use a multi-channel method to increase the transmission bandwidth, so as to achieve a high frame rate. In this embodiment, the frame rate can be, but is not limited to, up to 3000fps, and can be mainly used in the field of machine vision and industrial intelligent detection. In addition, the pixel format of the received image can be 8-bit data, or 10-bit or 12-bit RGB color data.

[0030] S102, writing and reading pixel data of each channel through a ping-pong operation of a single dual-port access memory, and performing parallel pixel expansion on edge pixels of each channel during the reading process; It should be noted first that the ping-pong operation mainly realizes seamless, high-speed caching and processing of data by alternating the storage and transmission of data streams through two data buffer modules (such as dual-port RAM, single-port RAM, FIFO, etc.). Specifically, the data stream is first cached to the first data buffer module (such as RAM A). After the first buffer cycle is completed, the data stream begins to be cached to the second data buffer module (such as RAM B), and the data in the first buffer module is sent to the data processing module for processing. In the next buffer cycle, the roles of the two data buffer modules are swapped, that is, the data stream is cached to the first buffer module, and the data in the second buffer module is processed. This process is repeated continuously to achieve continuous and uninterrupted processing of data.

[0031] Figure 3 is a schematic diagram of performing a ping-pong operation on an original image in a dual-port access device according to an embodiment of the present application, such as Figure 3 As shown, in the solution of the present application, a single dual-port access memory (True Dual Port RAM) is used in each channel to perform ping-pong operation to achieve alternating writing and reading of data, wherein the first port PortA of the True Dual Port RAM is used to write pixel data, and the second port PortB of the True Dual Port RAM is used to read pixel data; In this embodiment, optionally, 1024 pixel data need to be cached, so the address length of the True Dual Port RAM is 2048. The dual-port access device True Dual Port RAM is divided into two segments AB in the middle, and segment B is read when segment A is written, and segment A is read when segment B is written, so that only one RAM can be used for one channel to implement ping-pong operation.

[0032] In an exemplary embodiment, the process of writing and reading pixel data of each channel through a ping-pong operation of a single dual-port access memory includes the following steps: S1, in the first buffering period, buffering the input pixel data of any channel to the first port portA; S2, in the second buffering period, performing port switching through the input data selection unit, buffering the input pixel data to the second port portB, and simultaneously reading out the input pixel data in the first port portA; S3, in the third buffering period, port switching is performed through the input data selection unit, the input data is switched back to the first port portA, and at the same time the input pixel data in the second port portB is switched back to the first port portA.

[0033] In this scheme, the seamless, high-speed caching and processing of raw image data is achieved through the alternating use of two data buffer modules. The pipelined data processing method improves the overall processing efficiency of the system. At the same time, the ping-pong operation simplifies the complexity of data flow control, making the system design simpler and more reliable.

[0034] In addition, considering that during the filtering process, edge pixels may only exist at the edge of a single channel rather than the edge of the entire image, if these edge pixels are directly filtered, the filtering effect may be inaccurate due to the lack of sufficient context information; therefore, zero padding or copy padding cannot be simply used. These simple filling methods may cause deviations in the filtering effect in the edge area, affecting the overall image processing quality.

[0035] In order to ensure the effect of multi-channel filtering, it is necessary to expand the edge pixels with real data suitable for multi-channels. In this embodiment, when reading the True Dual Port RAM, the pixels on the left and right sides are repeatedly read, thereby supporting the expansion of the pixels on the left and right edges, so as to provide sufficient context information for these pixels during the filtering process. Through this processing method, it can be ensured that the filtering algorithm can also maintain high accuracy and consistency when processing edge pixels, thereby improving the overall image processing effect.

[0036] In an exemplary embodiment, referring to FIG. 2, taking the current channel N as an example, pixel expansion includes the following steps: Extension of the left pixel of the upper half of channel N: fill the pixels on the right edge of the upper half of channel N-1 to the left edge of the upper half of channel N; Extension of the right pixel of the upper half of channel N: fill the pixels on the left edge of the upper half of channel N+1 to the rightmost edge of channel N; In addition, it should be noted that the multi-channel image output by the CMOS image sensor in this embodiment is scanned in a special way, the upper channel is scanned from top to bottom, and the lower channel is scanned from bottom to top. Therefore, the two middle pixels of the image are exactly the lower edges of the upper and lower channels; therefore, the lower edge of the current channel N is not the real image edge, so it also needs to be expanded, such as Figure 2 As shown: The lower edge extension of the upper half of channel N needs to fill the lower edge pixels of the lower half of channel N.

[0037] Finally, reference Figure 2As shown in the figure, since the upper edge of each channel is also the upper edge of the image, such edges do not need special processing and can be filled with zeros. The left edge of the leftmost channel 0 upper half and the left edge of the channel 0 lower half are also the left edge of the image, and the right edge of the rightmost channel 31 upper half and the right edge of the channel 31 lower half are also the right edge of the image, so such edges do not need special processing and can be filled with zeros.

[0038] S103, through multiple sliding filter windows, parallel filtering is performed on the pixel data read from each channel according to a preset step size and a preset direction; weighted summation is performed on the filtered pixels in each window through a weighting matrix to obtain an image filtering result, wherein the weighting matrix is ​​dynamically formulated according to task requirements.

[0039] Figure 4 FIG. 1 is a schematic diagram of an overall processing flow of performing multi-channel parallel filtering on an original image according to an embodiment of the present application. Figure 4 As shown, for ease of understanding, taking three channels as an example, after acquiring the pixel data of each channel: Reading: The pixel data of each channel is read in parallel according to a specific sequence. Specifically, the rightmost pixel is read first, then 64 pixels are read in sequence from left to right, and then the leftmost pixel is read. It can be understood that after operating in the above manner, the left and right pixel data are read twice.

[0040] Pixel expansion: Because the pixels of the complete image and the image within the channel are scanned from left to right, the leftmost pixel of channel 1 needs to be filled with the rightmost pixel of channel 0, and the rightmost pixel of channel 1 needs to be filled with the leftmost pixel of channel 2. It can be understood that scanning a column of pixels in any channel requires 1+64+1, a total of 66 clock cycles. At the rising edge of the first clock, each channel fills the data into the right image channel, and does not fill at the rising edges of the 2nd to 65th clocks. At the rising edge of the 66th clock, the data of each channel is filled into the left image channel; according to this process, parallel pixel expansion is performed while reading each channel in parallel, so that the pixel expansion of the left and right edges is completed.

[0041] Filtering: The image data is processed through multiple 3x3 sliding filter windows. This step is implemented on the FPGA. By using the parallel processing capability of the FPGA, the filter windows can be efficiently slid and the pixels in each window can be weighted summed. Specifically, the implementation of the sliding window can be controlled by the internal logic of the FPGA, and it moves on the image according to the set step size and direction to ensure that each pixel is properly processed. It should be noted that since the core focus of the present application is the parallel processing of multi-channel pixels to achieve the optimization of processing efficiency; and the specific filtering logic of the sliding window has no effect on the invention of the present application, it will not be repeated in this embodiment.

[0042] In addition, in response to the filtering requirements in different scenarios, this application also screenshots the dynamic change of the weight matrix, so as to dynamically adjust the weight of each pixel according to the actual task requirements, and thus flexibly adapt to the filtering requirements in different scenarios; Specifically, this embodiment mainly includes two weighting matrices, a common Gaussian matrix, as shown below:

[0043] The other is an asymmetric matrix with high vertical weights and low horizontal weights, as shown below:

[0044] Among them, the Gaussian matrix can be applied to most common images, while the asymmetric matrix can retain more edge information in the horizontal direction and is more suitable for capturing images containing line lasers.

[0045] After the pixel data after filtering in each window is weighted, the final filtering result of the pixel is obtained by dividing it by 256. It can be understood that in the parallel filtering process, the flexible weighting matrix modification function provided by the present application solution allows users to adjust the parameters of the weighting matrix according to the specific usage scenario of the CMOS image sensor to meet the image processing requirements in different application scenarios. This flexibility enables the present invention to be widely applicable to various environments and tasks, further improving its practicality and adaptability.

[0046] Finally, it should be noted that after parallel filtering, the pixel data filled on both sides need to be deleted. In FPGA, it is not necessary to actually delete the data, but only to turn off the corresponding enable signal. Specifically, in this embodiment, the AXIS bus is used as the format for image transmission, and the redundant pixel data on both sides can be removed by simply pulling the axis_tvalid signal low.

[0047] Through the above step S103, the filtered result of each channel will replace the pixel value of the corresponding position in the original channel, or serve as the input of the subsequent image processing step. These results will maintain the independence between channels. If there is a demand, data merging or further processing between multiple channels can also be performed in subsequent steps.

[0048] Through the above steps S101 to S103, compared with the filtering method of the CMOS image sensor in the traditional technology, the present application scheme adopts optimization means such as FPGA technology and ping-pong operation to achieve real-time and efficient processing of the output data of the high frame rate CMOS image sensor, which can ensure that the processing efficiency of the CMOS image data matches the shooting speed of the front-end sensor; In addition, the multi-channel parallel filtering module is just suitable for the multi-channel CMOS image sensor. The preprocessing process, namely the filtering process, adopts a pipeline method to achieve efficient utilization of the performance of the multi-channel CMOS image sensor. Moreover, since the line structured light 3D camera often only needs to detect a certain type of object, it can also be adapted to different usage scenarios by appropriately adjusting the parameters of the 3x3 weighted matrix in the filtering to meet the filtering needs in different scenarios, thereby improving flexibility.

[0049] In summary, the present application not only solves the problem that the traditional image processing system cannot keep up with the speed, but also ensures the real-time, accuracy and flexibility of the image filtering processing tasks, providing a new solution for the fields of machine vision and industrial intelligent detection.

[0050] The embodiment of the present application also provides a parallel filtering system for a high frame rate CMOS image sensor. Figure 5 is a structural block diagram of a parallel filtering system of a high frame rate CMOS image sensor according to an embodiment of the present application, such as Figure 5 As shown, the system is implemented by a field programmable gate array, including: a collection module 50, an access module 51 and a filtering module 52, wherein: The acquisition module 50 is used to acquire the original images of multiple channels output by the CMOS image sensor; The access module 51 is used to write and read the pixel data of each channel through a ping-pong operation of a single dual-port access device, and in the process of reading, parallel pixel expansion is performed on the edge pixels of each channel; The filtering module 52 is used to perform parallel filtering processing on the pixel data read from each channel according to a preset step size and a preset direction through multiple sliding filtering windows, and to perform weighted summation on the filtered pixel data in each window through a weighting matrix to obtain an image filtering result, wherein the weighting matrix is ​​dynamically formulated according to task requirements.

[0051] Through this system, compared with the filtering method of CMOS image sensors in traditional technologies, this application scheme adopts optimization methods such as FPGA technology and ping-pong operation to achieve real-time and efficient processing of high frame rate CMOS image sensor output data. The multi-channel parallel filtering module is just suitable for multi-channel CMOS image sensors, and the calculation process is all pipelined, which realizes the efficient use of the performance of multi-channel CMOS image sensors; and because line structured light 3D cameras often only need to detect a certain type of object, the parameters of the 3x3 weighted matrix in the filter can also be appropriately adjusted according to different usage scenarios, so as to adapt to the filtering needs in different scenarios and improve flexibility.

[0052] In one embodiment, Figure 6 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 6 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 6 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system, the computer program is executed by the processor to implement a parallel filtering method for a high frame rate CMOS image sensor, and the database is used to store data.

[0053] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0054] In addition, in combination with the parallel filtering method of the high frame rate CMOS image sensor in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by the processor, any low-quality image target detection method in the above embodiment is implemented.

[0055] In one embodiment, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, the parallel filtering method for a high frame rate CMOS image sensor in any one of the above embodiments is implemented.

[0056] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0057] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0058] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A parallel filtering method for a high frame rate CMOS image sensor, characterized in that: Implemented by a field programmable gate array, the method comprises: Collecting raw images of multiple channels output by CMOS image sensors; The pixel data of each channel is written and read through a ping-pong operation of a single dual-port access device, and in the process of reading, parallel pixel expansion is performed on the edge pixels of each channel; Through multiple sliding filter windows, the pixel data read from each channel are filtered in parallel according to a preset step size and a preset direction, and the filtered pixel data in each window are weighted and summed through a weighting matrix to obtain an image filtering result, wherein the weighting matrix is ​​dynamically formulated according to task requirements.

2. The method according to claim 1, characterized in that: Any of the dual-port access devices is equally divided into a first port and a second port, and the first port and the second port implement pipeline writing and reading of pixel data in one channel through alternating ping-pong operations. During reading, edge pixels of each channel are repeatedly read to support edge pixel expansion of adjacent channels.

3. The method according to claim 2, characterized in that The pixel data for each channel is written and read via a ping-pong operation of a single dual-port accessor, including: Acting in a first buffering cycle, buffering the input pixel data of any channel to the first port; In a second buffering cycle, the input pixel data is cached to the second port and the input pixel data in the first port is read out at the same time by performing port switching through the input data selection unit; In a third buffering cycle, the input data selection unit performs port switching to switch the input data back to the first port, and simultaneously switches the input pixel data in the second port; The steps of the first buffer cycle, the second buffer cycle and the third buffer cycle are repeatedly cycled until all pixel data of any channel are read out, wherein edge pixels of each channel are repeatedly read during reading to support edge pixel expansion of adjacent channels.

4. The method according to claim 1, characterized in that: The field programmable gate array collects the original image through N channel data, and the upper half image and the lower half image of the original image occupy the upper half channel and the lower half channel of the N channels respectively.

5. The method according to claim 4, characterized in that During the reading process, performing parallel edge pixel expansion on edge pixels of each channel includes: For a first target channel in any non-image overall edge region: respectively obtaining first pixel data, second pixel data and third pixel data of left, right and bottom adjacent channels of an upper half channel of the first target channel; Fill the right pixel points of the first pixel data to the left edge of the first target channel, fill the left pixel points of the second pixel data to the right edge of the first target channel, fill the lower edge pixel points of the third pixel data to the lower edge of the first target channel, and perform zero filling on the upper edge of the target channel; For the second target channel located in the left edge area of ​​the entire image and the third target channel located in the right edge area, zero padding is performed on the left and right sides respectively; Zero pad the upper edge of each channel.

6. The method according to claim 5, characterized in that The method further comprises: Read the pixel data of each channel according to a specific sequence, wherein the specific sequence includes: At the beginning of the clock cycle, the rightmost pixel is read; In the middle clock cycle, the pixel data of each channel is read sequentially from left to right; At the end of the clock cycle, the leftmost pixel is read; At the rising edge of the start clock cycle, the rightmost pixel data of each channel other than the third target channel is filled to the left edge of the right adjacent channel, At the rising edge of the end clock cycle, the leftmost pixel data of each channel other than the second target channel is filled to the rightmost side of the adjacent channel.

7. The method according to claim 1, characterized in that The weighting matrix includes a Gaussian matrix and an asymmetric matrix. Among them, the parameters of the weighting matrix are dynamically adjusted according to the requirements of the filtering task, the filtered pixel data in each filtering window are weighted summed up respectively through the dynamically adjusted weighting matrix, and the weight of each pixel point in the weighted summation result is adjusted to obtain image filtering results suitable for different scenarios.

8. A parallel filtering system for a high frame rate CMOS image sensor, characterized in that: The system is implemented by a field programmable gate array and includes: an acquisition module, an access module and a filtering module, wherein: The acquisition module is used to acquire original images of multiple channels output by the CMOS image sensor; The access module is used to write and read the pixel data of each channel through a ping-pong operation of a single dual-port access device, and in the reading process, parallel pixel expansion is performed on the edge pixels of each channel; The filtering module is used to perform parallel filtering processing on the pixel data read from each channel according to a preset step size and a preset direction through multiple sliding filtering windows, and to perform weighted summation on the filtered pixel data in each window through a weighting matrix to obtain an image filtering result, wherein the weighting matrix is ​​dynamically formulated according to task requirements.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Data processing and transmitting system of high-speed multichannel CCD (charge-coupled device)

    CN102638661A

  • Parallel real-time SAR image spot and noise reducing method based on multiple DSPs

    CN103761709A

  • Multi-core digital signal processor (DSP) based multi-channel image acquisition processing system

    CN105516624A

  • Picture filtering method and device

    CN110062225A

  • Micro-ringing blocking rapid Wiener restoration method for single remote sensing image

    CN111127330A

Cited By

  • Image data amplification processing method and device, electronic equipment and storage medium

    CN121616452A

  • Magnification processing method and device of image data, electronic equipment and storage medium

    CN121616452B