Image data processing method and sensor device

Abstract

An image data processing method and a sensor device are disclosed. The sensor device includes: an image sensor configured to acquire image data; an image buffer configured to store the image data; and an image processor configured to generate image-processed data by applying a filter corresponding to the storage mode of the image buffer to the image data stored in the image buffer.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0011153, filed with the Korean Intellectual Property Office on January 30, 2020, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to image data processing. Background Art

[0003] Various types of image data processing techniques, including techniques for changing an image represented by data to suit a purpose, are actively used and are used for image manipulation, image analysis, image recognition, and image communication. Among various image data processing techniques, convolution processing is widely used for blurring effects or sharpening effects. In one example, convolution processing is a processing method of multiplying pixel values of a central pixel and its adjacent pixels in image data by weights (or coefficients) assigned to a filter (or mask), and assigning a result value corresponding to their sum as the pixel value of the central pixel. Summary of the Invention

[0004] The present invention content is provided to introduce a selection of concepts that will be further described in the detailed description below in a simplified form. The present invention content is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to assist in determining the scope of the claimed subject matter.

[0005] In one general aspect, there is provided a sensor device including: an image sensor configured to acquire image data; an image buffer configured to store the image data; and an image processor configured to generate image-processed data by applying a filter corresponding to a storage mode of the image buffer to the image data stored in the image buffer.

[0006] The image processor may be configured to select a filter corresponding to the storage mode of the image buffer from among a plurality of filters of different modes, and apply the selected filter to the image data stored in the image buffer.

[0007] The number of filters of different modes may be determined based on the number of rows of image data that can be stored in the image buffer and the size of the filter.

[0008] The image processor may be configured to select a filter of a mode for the image data currently stored in the image buffer from among a plurality of filters of a plurality of modes based on a buffer region of the image buffer in which the most recently stored image data is stored.

[0009] The image processor may be configured to generate, as image - processed data, image data of convolution processing by applying a convolution filter for convolution processing to the image data stored in the image buffer.

[0010] The convolution processing for the image data stored in the image buffer may be performed in the sensor device.

[0011] The image processor may be configured to, in response to a row of image data being input and stored in the image buffer, output a row of the result image by applying a convolution filter to the image data stored in the image buffer.

[0012] The image buffer may be configured to store image data of pixels including a plurality of color components arranged in a pattern, and the image processor may be configured to generate image - processed data by applying a filter corresponding to the pattern to the image data stored in the image buffer.

[0013] The image processor may be configured to apply a first filter corresponding to the arrangement pattern of the first color component in the pattern to the image data stored in the image buffer, and apply a second filter corresponding to the arrangement pattern of the second color component in the pattern to the image data stored in the image buffer.

[0014] The first color component and the second color component may be different color components, and the first filter and the second filter are alternately applied to the pixels of the image data stored in the image buffer.

[0015] The image buffer may be configured to sequentially store a portion of the image data acquired by the image sensor in units of rows.

[0016] The image processor may be configured to apply a filter to the image data stored in the image buffer, and generate quantized image data by quantizing the image data to which the filter has been applied.

[0017] The sensor device may include an output interface configured to send the image - processed data to the outside of the sensor device.

[0018] The image buffer may be configured to sequentially overwrite rows of the image data.

[0019] The image processor may be configured to select a filter for the image data currently stored in the image buffer from a plurality of filters of a plurality of patterns based on the most recently changed row of the image data in the image buffer.

[0020] The output interface may be configured to send the image - processed data to the outside of the sensor device in response to an object being detected in the image - processed data.

[0021] In another general aspect, there is provided an image data processing method performed by a sensor device, the image data processing method comprising: acquiring image data using an image sensor; storing the acquired image data in an image buffer; and generating processed image data by applying a filter corresponding to the storage mode of the image buffer to the image data stored in the image buffer.

[0022] The step of generating processed image data may include: selecting a filter corresponding to the storage mode of the image buffer from a plurality of filters of different modes, and applying the selected filter to the image data stored in the image buffer.

[0023] The step of generating processed image data may include: generating convolution-processed image data as the processed image data by applying a convolution filter for convolution processing to the image data stored in the image buffer.

[0024] The step of generating processed image data may include: in response to a row of the acquired image data being input and stored in the image buffer, generating a row of a result image by applying a convolution filter to the image data stored in the image buffer.

[0025] The image data may include pixels of a plurality of color components arranged in a pattern, and the generating step may include: generating processed image data by applying a filter corresponding to the pattern to the image data stored in the image buffer.

[0026] The generating step may include: applying a first filter corresponding to the arrangement pattern of the first color component in the pattern to the image data stored in the image buffer; and applying a second filter corresponding to the arrangement pattern of the second color component in the pattern to the image data stored in the image buffer.

[0027] In another general aspect, there is provided a sensor device, the sensor device comprising: an image sensor configured to acquire image data; an image buffer configured to sequentially store image data in units of rows; and an image processor configured to: process the stored image data by applying a filter corresponding to the storage mode of the row of the image data most recently stored in the image buffer.

[0028] Other features and aspects will be apparent from the following detailed description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 An example showing the overall configuration of the recognition system.

[0030] Figure 2 Shows an example of object recognition processing.

[0031] Figure 3 Shows an example of the operations of an image data processing method performed by a sensor device.

[0032] Figure 4 Shows an example of a convolution processing procedure.

[0033] Figure 5 and Figure 6 Shows an example of performing convolution processing on image data.

[0034] Figures 7A to 8 Shows an example of performing convolution processing on image data in a Bayer pattern.

[0035] Figure 9 Shows an example of the configuration of a sensor device.

[0036] Figure 10 Shows an example of the configuration of an identification device.

[0037] Throughout the drawings and the detailed description, unless otherwise described or provided, the same reference numerals will be understood to represent the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Description

[0038] The following detailed description is provided to assist the reader in obtaining a comprehensive understanding of the methods, devices, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be apparent after understanding the disclosure of this application. For example, the order of operations described herein is merely exemplary and is not limited to the order of operations set forth herein. Rather, the order of operations may be changed as will be apparent after understanding the disclosure of this application, except for operations that must occur in a particular order. Additionally, descriptions of features known in the art may be omitted for greater clarity and conciseness.

[0039] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Instead, the examples described herein are provided only to illustrate some of the many possible ways of implementing the methods, devices, and / or systems described herein that will be apparent after understanding the disclosure of this application.

[0040] Hereinafter, examples will be further described with reference to the drawings. The following structural descriptions or functional descriptions of the examples disclosed in this disclosure are only for the purpose of describing the examples, and the examples may be implemented in various forms. The examples are not meant to be limiting, but are intended to cover various modifications, equivalents, and alternatives within the scope of the claims as well.

[0041] Terms such as first, second, etc. may be used herein to describe components. Each of these terms is not used to define the nature, order, or sequence of the corresponding component, but only to distinguish the corresponding component from other components. It should be noted that if a component is described as "connected", "coupled", or "joined" to another component, although the first component may be directly connected, coupled, or joined to the second component, a third component may be "connected", "coupled", or "joined" between the first component and the second component.

[0042] Unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. It will also be understood that when the terms "comprises" and / or "comprising" are used herein, it states the presence of the stated features, wholes, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof.

[0043] Hereinafter, examples will be described in detail with reference to the drawings. When describing examples with reference to the drawings, the same reference numerals denote the same constituent elements, and the repeated descriptions related thereto will be omitted.

[0044] Figure 1 An example showing the overall configuration of the recognition system is shown.

[0045] Referring to Figure 1 , the recognition system 100 includes a sensor device 110 and a recognition device 120. The recognition system 100 acquires image data through the sensor device 110, and recognizes an object shown in the image data through the recognition device 120. The recognition system 100 is used for object recognition (such as face recognition or recognition of things).

[0046] The sensor device 110 acquires image data using an image sensor (such as a camera), and generates processed image result data by performing image processing on the acquired image data. The sensor device 110 generates convolution processing result data by performing convolution processing on the acquired image data. The sensor device 110 performs convolution processing by applying a filter for performing convolution operation to the image data. If the image data acquired through the image sensor is Bayer pattern image data, the sensor device 110 also generates convolution processing result data by performing convolution processing on the Bayer pattern image data. As described above, the convolution processing of the image data is performed in the sensor device 110. Here, the term "filter" may be used interchangeably with the terms mask, kernel, or template.

[0047] In another example, the sensor device 110 compresses the image data that has been convolutionally processed. For example, the sensor device 110 generates quantized image data as result data by quantizing the convolutionally processed image data. The quantized image data is image data that has fewer bits than the original image data acquired by the image sensor (e.g., binary image data in which each pixel among all pixels of the image data has one of a first pixel value (e.g., "0") and a second pixel value (e.g., "1")).

[0048] In one example, the sensor device 110 autonomously performs image processing (such as convolutional processing and / or quantization) using an internal processing unit (e.g., an image processor or a microcontroller unit). The result data after the image processing is sent to the recognition device 120. Here, the image processor refers to a processor that performs the function of processing image data in the sensor device 110 and should not be construed as being limited to a processor that only performs the function of processing images. In one example, the image processor performs functions other than image processing (e.g., controlling the sensor device 110).

[0049] As described above, image processing is first performed on the image data acquired by the sensor device 110 in the sensor device 110 rather than in the recognition device 120. For example, the sensor device 110 performs convolutional processing and / or quantization on the image data. In this example, the result data after the image processing is sent outside the sensor device 110, and thus, the bandwidth and the amount of data transmission required can be reduced.

[0050] The sensor device 110 autonomously performs processing operations that require relatively low computational complexity. For example, the sensor device 110 performs processing operations such as object detection (e.g., face detection) after performing convolutional processing and / or quantization on the acquired image data. If an object is detected in the image data, the sensor device 110 sends the convolutionally processed image data and / or the quantized image data outside the sensor device 110, and if an object is not detected in the image data, the sensor device 110 does not send the image data outside. In this example, the convolutionally processed image data and / or the quantized image data are sent from the sensor device 110 only when an object is detected in the image data.

[0051] The recognition device 120 receives the result data of image processing performed by the sensor device 110 from the sensor device 110, and performs object recognition based on the received result data. In one example, the recognition device 120 uses a trained object recognition model to perform object recognition. The object recognition model is, for example, a neural network model, and provides a score (e.g., an expected value or a probability value) indicating that the object corresponds to the image data shown in the image data input to the object recognition model. The object recognition model is a neural network model having a bit width corresponding to the bit width of the result data received from the sensor device 110. If the result data received from the sensor device 110 is quantized image data with a low bit width, a neural network model with a corresponding low bit width is used, so that the amount of required resources is reduced and the processing rate is increased. By using the object recognition model with a low bit width as described above, the recognition system 100 performs object recognition processing with low power, less memory capacity usage, and high speed in a limited embedded system (such as an intelligent sensor or a smart phone).

[0052] Figure 2 Shows an example of object recognition processing. Figure 2 The operations in can be performed in the order and manner shown in the figure. However, some operation orders can be changed or some operations can be omitted without departing from the spirit and scope of the described illustrative examples. Figure 2 Many of the operations shown in can be performed in parallel or simultaneously. Figure 2 The blocks of the object recognition processing and the combination of the blocks are executed by the object recognition device. In one example, the object recognition device is implemented by a computer based on dedicated hardware and a device (such as a processor) that executes a specified function, or a combination of dedicated hardware and computer instructions included in the object recognition device. Except for the following Figure 2 description, Figure 1 the description of Figure 2 also applies to

[0053] Refer to Figure 2 , the object recognition processing includes operation 210 and operation 220. In operation 210, the sensor device generates processed result data by performing image processing on the image data. In operation 220, the recognition device performs object recognition based on the processed result data.

[0054] In operation 210, a sensor device (in operation 212) acquires image data using an image sensor and (in operation 214) stores the acquired image data in an image buffer in the sensor device. In one example, the image buffer included in the sensor device has a limited capacity. Thus, the image buffer stores portions of the image data acquired by the image sensor sequentially, rather than storing the entire image data. For example, the image buffer stores image data of several (e.g., 5) rows (e.g., rows) among the multiple rows that make up the image data at a single point in time. Here, a row refers to a group of pixels arranged along the same horizontal direction or the same vertical direction among the multiple pixels included in the image data. In one example, due to the limited buffer space, the image buffer stores the image data in units of rows and stores the image data in a manner that sequentially overwrites the image data. For example, assuming that the image buffer stores only 5 rows of image data due to the limited buffer area, whenever the image buffer receives a new row of image data, the newly received row of image data is stored in the row in the buffer area of the image buffer where the image data that has been stored for the longest time is located. Thus, the image data that has been recorded for the longest time among the image data stored in the buffer area of the image buffer is removed from the buffer area.

[0055] The image data acquired by the image sensor has a pattern (e.g., Bayer pattern) in which pixels of multiple color components are arranged, and the image data of this pattern is stored in the image buffer. The general Bayer pattern includes pixel values of different color components, e.g., pixel values of a green component, a blue component, and a red component, where, according to the row, the pixel values of the blue component and the green component are arranged alternately, or the pixel values of the red component and the green component are arranged alternately. However, the Bayer pattern of the image data that can be acquired by the image sensor is not limited to a specific Bayer pattern, and there can be various arrangement patterns of the pixel values of the color components that make up the image data. For example, the image data has a tetracell pattern or a britecell pattern. In the tetracell pattern, pixel values of the same color component are set as a single group. In the britecell pattern, the pixel value of the green component in the Bayer pattern is replaced with a white component pixel value. Hereinafter, for ease of description, a description will be provided based on the general Bayer pattern. However, the pattern of the image data that can be acquired by the image sensor is not limited to the general Bayer pattern, and there can be an unlimited number of types of patterns.

[0056] In operation 216, the sensor device performs image processing on the image data stored in the image buffer, and in operation 218, the sensor device generates the processed image data as result data. The sensor device performs image processing such as convolution processing using a mode corresponding to the storage mode of the image buffer according to the time point at which the image processing is performed. As described above, the storage mode of the image data stored in the image buffer changes over time due to sequential storage, and the sensor device performs convolution processing using a filter of a mode corresponding to the storage mode of the image data stored in the image buffer among a plurality of filters. Therefore, the mode of the filter applied to the image data in the image buffer also changes over time, and the change of the mode repeats periodically.

[0057] In some examples, quantization may be further performed after performing convolution processing on the image data. In one example, during quantization, the sensor device uses a halftone technique to quantize the image data. The halftone technique is a technique for converting image data with a high bit width into image data with a low bit width (e.g., binary image data), where the pixel values of the image are represented as densities in a two-dimensional space. For example, the sensor device quantizes the image data by applying a quantization filter (such as a dither matrix) to the image data, or uses error diffusion to quantize the image data.

[0058] The processed result data generated by the sensor device is sent to the recognition device through the output interface of the sensor device. For example, the sensor device sends the result data to the recognition device through an output interface (such as a Mobile Industry Processor Interface (MIPI)). The number of rows of the result data sent from the sensor device to the recognition device is determined based on the MIPI standard. The recognition device includes, for example, an application processor (AP) connected to the MIPI.

[0059] In operation 220, the recognition device performs object recognition using the object recognition model 222. The object recognition model 222 is, for example, a trained neural network model, and provides a recognition result regarding object recognition based on the processed image data received from the sensor device. The object recognition process is performed under the control of a processor, which may be a hardware-implemented data processing device having a circuit physically configured to perform a desired operation (such as a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU) included in the recognition device). If image data is input into the object recognition model 222, the object recognition model 222 outputs a score indicating the probability or likelihood that the object shown in the image data corresponds to each class (or label). The recognition device determines the object recognition result based on the information related to the score.

[0060] Figure 3 An example of an operation of an image data processing method performed by a sensor device is shown. Figure 3 The operations in can be performed in the order and manner shown in the figure. However, the order of some operations can be changed or some operations can be omitted without departing from the spirit and scope of the described illustrative example. Figure 3 Many of the operations shown in can be performed in parallel or simultaneously. Figure 3 The blocks and combinations of blocks of the image data processing method of are performed by a computer based on dedicated hardware and a device (such as a processor) that executes a specified function or a combination of dedicated hardware and computer instructions included in the sensor device. Except for the following Figure 3 description, Figures 1 to 2 the description of also applies to Figure 3 and is incorporated herein by reference. Therefore, the above description may not be repeated here.

[0061] Referring to Figure 3 , in operation 310, the sensor device uses an image sensor (such as a camera) to acquire image data. In operation 320, the sensor device stores the acquired image data in an image buffer. Due to the limited buffer area of the image buffer, the sensor device sequentially stores portions of the image data in units of rows. In another example, the image sensor acquires image data in a Bayer pattern, and the image data in the Bayer pattern is stored in the image buffer.

[0062] In operation 330, the sensor device generates processed result data by performing image processing on the image data stored in the image buffer. The sensor device generates the processed result data by applying a filter corresponding to the current storage mode of the image buffer to the image data stored in the image buffer. For example, the sensor device generates image data of convolution processing as result data by applying a convolution filter for convolution processing to the image data stored in the image buffer. When the rows of the image data acquired by the image sensor are input and stored in the image buffer, the sensor device performs convolution processing by applying the convolution filter to the image data stored in the image buffer and outputs the rows of the result image. The convolution filter includes coefficients or weights to be applied to the image data, and the pixel values of the result image of the convolution processing using the convolution filter are determined based on the pixel values of the image data stored in the image buffer. As described above, convolution processing for the image data stored in the image buffer is performed in the sensor device.

[0063] The sensor device selects a filter corresponding to the current storage mode of the image buffer from filters of different modes, and applies the selected filter to the image data stored in the image buffer. In one example, the number of filters of different modes is determined based on the number of rows of image data that can be stored in the image buffer and the size of the filter. For example, if the image buffer can store five rows of image data for the buffer size of the image buffer and the size of the filter is 5×5, there may be five filters of different modes.

[0064] The sensor device selects a filter of the mode to be applied to the image data currently stored in the image buffer based on the buffer region in which the image data was most recently stored in the image buffer. Based on the most recently changed row among the rows included in the buffer region of the image buffer, a filter to be used for image processing is determined from among the filters of multiple modes.

[0065] When the image data is stored in the entire buffer region of the image buffer, or when the image data is stored in a part of the entire buffer region, an image processing process using the filter begins. If the image data is stored in a part of the entire buffer region, image processing is performed using zero-padding or wrap-around. Zero-padding performs image processing by filling the parts required for image processing with the value "0", and wrap-around processes the image data as if the start part and the end part of the image data are connected to each other.

[0066] If the image data stored in the image buffer is image data in a pattern in which pixels having a plurality of color components are arranged (e.g., Bayer pattern), the sensor device generates processed image result data by applying a filter corresponding to the arrangement pattern of the color components in the Bayer pattern of the image data stored in the image buffer to the image data stored in the image buffer. For example, at a first time point with respect to the image data stored in the image buffer, the sensor device applies a first filter corresponding to the arrangement pattern of a first color component (e.g., blue component) in the Bayer pattern to the image data stored in the image buffer, and applies a second filter corresponding to the arrangement pattern of a second color component (e.g., green component) different from the first color component to the image data stored in the image buffer. The first filter and the second filter are alternately applied to the pixels of the image data stored in the image buffer. For example, at a second time point with respect to the image data stored in the image buffer, the sensor device applies the second filter corresponding to the arrangement pattern of the second color component (e.g., green component) in the Bayer pattern to the image data stored in the image buffer, and applies a third filter corresponding to the arrangement pattern of a third color component (e.g., red component) different from the second color component to the image data stored in the image buffer. The second filter and the third filter are also alternately applied to the pixels of the image data stored in the image buffer.

[0067] In some examples, the sensor device generates result data with a low bit width by performing quantization after convolution processing. The sensor device generates quantized image data by applying a quantization filter to the image data on which convolution processing is performed. The image data acquired by the image sensor has a large number of bits. The sensor device reduces the number of bits of the image data corresponding to the result data by autonomously performing quantization via a processing unit (such as an image processor or a microcontroller unit) included in the sensor device.

[0068] In operation 340, the sensor device sends the processed image result data to the outside of the sensor device. The sensor device sends the result data to the outside of the sensor device through an output interface (such as MIPI). As described above, the image data acquired by the image sensor is output from the sensor device after being processed in the sensor device.

[0069] Figure 4 An example of the convolution processing procedure is shown. Figure 4 The operations in can be performed in the order and manner as shown in the figure. However, the order of some operations can be changed or some operations can be omitted without departing from the spirit and scope of the described illustrative examples. Figure 4 Many of the operations shown in can be performed in parallel or simultaneously. Figure 4The blocks and combinations of blocks of the convolution processing process are executed by a dedicated hardware-based computer and a device (such as a processor) that performs a specified function, or a combination of dedicated hardware and computer instructions included in the convolution processing device. Except for the following Figure 4 description, Figures 1 to 3 the description also applies to Figure 4 and is incorporated herein by reference. Therefore, the above description will not be repeated here.

[0070] Referring to Figure 4 , in operation 410, the current line of image data is received through the image buffer. In operation 420, the sensor device stores the current line of the received image data in the image buffer. For example, the sensor device records the current line of image data in the buffer area of the image buffer that stores the oldest image data. Here, the buffer area for recording image data is defined in units of lines.

[0071] In operation 430, the sensor device performs convolution processing using a filter corresponding to the storage mode of the image data in the image buffer. For example, the sensor device performs image processing using a convolution filter corresponding to the storage mode of the image data currently stored in the image buffer among the predefined mode convolution filters. Here, the storage mode of the image data changes, for example, according to the line that stores the most recent image data among multiple lines included in the buffer area of the image buffer.

[0072] In operation 440, the sensor device outputs the result data of the convolution processing. The sensor device outputs the line of the result image as the result data through convolution processing by applying the convolution filter to the image data stored in the image buffer.

[0073] In operation 450, the sensor device determines whether the image processing for the entire image data is completed. The sensor device determines whether the last line of the image data acquired by the image sensor has been sent and stored in the image buffer and whether the convolution processing for the last line of the image data is all completed. If the image processing for the entire image data is all completed by completing the image processing for the last line of the image data, the convolution processing process is terminated.

[0074] If the image processing for the entire image data has not been completed, in operation 460, the next line of the image data is received through the image buffer. In one example, the next line of the image data is the line immediately following the current line of the image data received in operation 410 in the image data acquired by the image sensor. In operation 470, the sensor device stores the next line of the received image data in the image buffer. The sensor device records the next line of the image data in the buffer area of the image buffer that stores the oldest image data. The sensor device starts executing the image processing procedure again from operation 430.

[0075] As described above, the sensor device sequentially stores the image data acquired by the image sensor in the image buffer in units of lines, and performs convolution processing using a filter corresponding to the storage mode of the image data stored in the image buffer, so that the convolution processing is effectively performed in the sensor device regardless of the limited buffer area of the image buffer.

[0076] Figure 5 and Figure 6 illustrates an example of performing convolution processing on image data.

[0077] Figure 5 illustrates that the image data 500 is acquired by the image sensor of the sensor device. The image data 500 includes a plurality of lines (e.g., 1, 2, 3,..., 10), and each line is a set of pixels. For example, the line "1" of the image data 500 is a set of pixels corresponding to the first line, and the line "2" of the image data 500 is a set of pixels corresponding to the second line. The image data 500 is stored in the image buffer included in the sensor device. In this example, it is assumed that the buffer area of the image buffer stores only five lines of image data. However, the size of the buffer area of the image buffer is not limited thereto.

[0078] Similar to a rolling shutter, the image buffer stores the image data 500 sequentially line by line. For example, the image buffer stores the lines "1" to "5" of the image data 500 in sequence, as shown in the storage pattern 510. Due to the limited buffer area, the image buffer stores the received image data in an overwriting manner. For example, if the line "6" of the image data 500 is received in the storage state of the image buffer shown in the storage pattern 510, the image buffer stores the line "6" of the image data by overwriting the line "6" of the image data into the oldest recorded buffer area (the buffer area recording the line "1" of the image data), as shown in the storage pattern 520. Thereafter, if the line "7" of the image data 500 is received, the image buffer records the line "7" of the image data into the oldest recorded buffer area (the buffer area recording the line "2" of the image data), as shown in the storage pattern 530. In this way, the storage patterns of the image data stored in the image buffer are sequentially represented in the order of the storage pattern 510, the storage pattern 520, the storage pattern 530, the storage pattern 540, the storage pattern 550, and the storage pattern 560. As described above, the image data 500 is sent line by line to the image buffer, and the image buffer stores the lines of the received image data in the buffer area in an overwriting manner.

[0079] Refer to Figure 6 , an example of performing convolution processing for each storage pattern of the image buffer is shown. When the storage pattern of the image buffer changes, the mode of the filter for convolution processing also changes according to the storage pattern of the image buffer. The sensor device selects a filter corresponding to the current storage pattern of the image buffer from filters of different modes 610, 620, 630, 640, and 650, and applies the selected filter to the image data stored in the image buffer.

[0080] For example, corresponding to the storage pattern 510 of the image data in the image buffer, the filter of pattern 610 is applied to the image data, and corresponding to the storage pattern 520 of the image data in the image buffer, the filter of storage pattern 620 is applied. In the case of storage pattern 510, the filter of pattern 610 as the basic pattern is directly applied. However, in the case of storage pattern 520, since the order of the rows of the image data stored in the image buffer is not sequential, the filter of pattern 610 as the basic pattern should not be directly applied. Therefore, the sensor device rotates the components of the filter to correspond to the order of storage pattern 520, as shown in pattern 620, and uses the filter of pattern 620 in which the components are arranged in the order of "5, 1, 2, 3, 4". The data corresponding to row "a" of the result data 690 (the pixel value of the convolution process) is determined by applying the filter of pattern 610 to the image data of storage pattern 510, and the data corresponding to row "b" of the result data 690 is determined by applying the filter of pattern 620 to the image data of storage pattern 520.

[0081] In addition, the filter of pattern 630, the filter of pattern 640, and the filter of pattern 650 are respectively applied corresponding to storage pattern 530, storage pattern 540, and storage pattern 550. If the row "7" of the image data is stored in the image buffer as shown in storage pattern 530, the order of the image data stored in the image buffer is "6, 7, 3, 4, 5". In this example, the sensor device uses the filter of pattern 630 having a component order "4, 5, 1, 2, 3" corresponding to the order of storage pattern 530.

[0082] In the case of storage pattern 560, the data corresponding to row "f" of the result data 690 is determined by applying the filter of pattern 610 again. As a cycle, the pattern of the filter applied to the image buffer changes repeatedly from pattern 610 to pattern 650. The filter having a pattern that changes periodically as described above will be called a "rotating filter". Whenever the image data is input to the image buffer, the pattern of the filter for the convolution process changes (for example, the components of the filter are rotated). The number of filters of different patterns corresponds to the number of rows of the image data that can be stored in the image buffer. For example, assuming that the number of rows of the image data that can be stored in the image buffer is "5", there are 5 filters of different patterns. The sensor device obtains the result data 690 of the convolution process by sequentially applying the filter corresponding to the storage pattern of the image buffer whenever a new row of the image data is input to the image buffer.

[0083] Figures 7A to 8 An example of performing convolution processing on image data in the Bayer pattern is shown.

[0084] Refer to Figure 7A , an example of Bayer pattern image data 700 obtained by an image sensor is shown. The Bayer pattern image data 700 is represented in a form in which pixel values of blue component 702, pixel values of green components 704 and 706, and pixel values of red component 708 are arranged. Color components corresponding to each pixel of the image data 700 are predefined. For example, odd rows of the image data 700 include pixel values of blue component 702 and pixel values of green component 704 arranged alternately, and even rows of the image data 700 include pixel values of green component 706 and pixel values of red component 708 arranged alternately. However, the Bayer pattern is not limited to this example and can have various forms.

[0085] Figure 7B An example of storing Figure 7A the Bayer pattern image data 700 is shown. Refer to Figure 7B , the image data 700 is stored in the buffer area of the image buffer sequentially row by row, so the storage pattern of the image data stored in the image buffer changes over time.

[0086] For example, the image buffer stores rows "1" to "5" of the Bayer pattern image data 700 sequentially as shown in storage pattern 710. Due to the limited buffer area, the image buffer stores the received image data in an overwriting manner. For example, if row "6" of the image data 700 is received in the storage state of the image buffer shown in storage pattern 710, the image buffer stores row "6" of the image data by overwriting row "6" of the image data into the buffer area that has been recorded for the longest time (the buffer area that records row "1" of the image data), as shown in storage pattern 715. If the image data 700 is stored in the image buffer in this way, the image data is stored in the image buffer in the order of storage pattern 710, storage pattern 715, storage pattern 720, storage pattern 725, storage pattern 730, storage pattern 735, storage pattern 740, storage pattern 745, storage pattern 750, storage pattern 755, storage pattern 760... as Figure 7B shown. Storage pattern 760 shows the same pattern as storage pattern 710, and storage patterns 710 to 755 repeat from storage pattern 760.

[0087] Figure 8An example of performing convolution processing on Bayer pattern image data stored in an image buffer in different modes is shown. When the Bayer pattern image data is stored in the image buffer, the storage mode of the image buffer changes in the order from storage mode 710 to storage mode 730. The sensor device generates processed result data 890 by applying a filter corresponding to the arrangement pattern of color components in the Bayer pattern of the image data stored in the image buffer to the image data stored in the image buffer. The result data 890 also has a Bayer pattern. The sensor device selects a filter corresponding to the current storage mode of the image buffer from filters of a predefined pattern and applies the selected filter to the image data stored in the image buffer.

[0088] The sensor device applies a filter corresponding to the arrangement pattern of color components in the Bayer pattern of the image data stored in the image buffer to the image data. For example, by alternately applying the filter 812 corresponding to the blue component and the filter 814 corresponding to the green component (or applying the filters 812 and 814 for every two spaces) to the image data of storage mode 710, convolution operation is performed on the image data of storage mode 710, and the data corresponding to row "a" of the result data 890 (the pixel value of the convolution processing) is determined. Here, the filter 816 corresponding to the red component is not applied. By alternately applying the filter 824 corresponding to the green component and the filter 826 corresponding to the red component to the image data of storage mode 715, convolution operation is performed on the image data of storage mode 715, and the data corresponding to row "b" of the result data 890 is determined. Here, the filter 822 corresponding to the blue component is not applied. By alternately applying the filter 832 corresponding to the blue component and the filter 834 corresponding to the green component to the image data of storage mode 720, convolution operation is performed on the image data of storage mode 720, and the data corresponding to row "c" of the result data 890 is determined. Here, the filter 836 corresponding to the red component is not applied. By alternately applying the filter 844 corresponding to the green component and the filter 846 corresponding to the red component to the image data of storage mode 725, convolution operation is performed on the image data of storage mode 725, and the data corresponding to row "d" of the result data 890 is determined. Here, the filter 842 corresponding to the blue component is not applied. By alternately applying the filter 852 corresponding to the blue component and the filter 854 corresponding to the green component to the image data of storage mode 730, convolution operation is performed on the image data of storage mode 730, and the data corresponding to row "e" of the result data 890 is determined. Here, the filter 856 corresponding to the red component is not applied.

[0089] As described above, whenever a new line of image data is input and stored in the image buffer, the sensor device obtains result data 890 by sequentially applying a filter of a pattern corresponding to the Bayer pattern of the image data stored in the image buffer. In this example, the sensor device performs a convolution operation for each color component, but does not apply the filter used for the convolution operation to the color components not included in the line of the output data 890.

[0090] Figure 9 An example of the configuration of the sensor device is shown.

[0091] Referring Figure 9 , the sensor device 900 includes an image sensor 910, an image processor 920, a memory 930, an image buffer 940, and an output interface 950. The sensor device 900 corresponds to the sensor device described herein.

[0092] The image sensor 910 acquires image data. For example, the image sensor 910 acquires image data such as a color image, a grayscale image, or an infrared image. The image buffer 940 stores the image data acquired by the image sensor 910. The image buffer 940 stores portions of the image data sequentially in units of rows.

[0093] The image processor 920 controls the operation of the sensor device 900 and includes, for example, any one or any combination of a digital signal processor (DSP), an image signal processor (ISP), and a microcontroller unit (MCU).

[0094] The image processor 920 performs image processing (such as convolution processing and / or quantization) on the image data stored in the image buffer 940. The image processor 920 performs one or more operations related to the image processing described above with reference to Figures 1 to 8 the description.

[0095] For example, the image processor 920 generates result data of the convolution processing by applying a convolution filter corresponding to the current storage mode of the image buffer 940 to the image data stored in the image buffer. If the image data stored in the image buffer is Bayer pattern image data, the image processor 920 generates processed image data as result data by applying a filter corresponding to the arrangement pattern of the color components in the Bayer pattern of the image data stored in the image buffer to the image data stored in the image buffer. In another example, the image processor 920 generates quantized image data as result data by performing quantization on the image data of the convolution processing.

[0096] The memory 930 stores instructions to be executed by the processor 920 and information to be used for performing image processing. The memory 930 stores image data acquired by the image sensor 910 and result data of the processed image. The memory 930 includes, for example, high-speed random access memory and / or non-volatile computer-readable storage media. Further details regarding the memory 930 are provided below.

[0097] The output interface 950 sends the result data of the image processing performed by the image processor 920 to the outside of the sensor device 900. The output interface 950 sends the result data to the recognition device through wired communication or wireless communication. For example, the output interface 950 is MIPI. In this example, the sensor device 900 sends the result data at a bandwidth determined by the MIPI standard.

[0098] Figure 10 An example of the configuration of the recognition device is shown.

[0099] Refer to Figure 10 , the recognition device 1020 includes: a processor 1030, a memory 1040, a storage device 1050, an input device 1060, an output device 1070, and a communication device 1080. Multiple elements of the recognition device 1020 communicate with each other through a communication bus. The recognition device 1020 corresponds to the recognition device described herein.

[0100] The processor 1030 controls the operation of the recognition device 1020 and executes instructions and functions to perform object recognition. For example, the processor 1030 executes instructions stored in the memory 1040 or the storage device 1050. The processor 1030 includes, for example, any one or any combination of a CPU, a GPU, and an NPU, and executes one or more of the operations related to the object recognition described with reference to Figures 1 to 9 . For example, the processor 1030 performs object recognition based on the processed image result data received from the sensor device 900.

[0101] The memory 1040 stores instructions to be executed by the processor 1030 and information to be used for performing object recognition. The memory 1040 includes, for example, high-speed random access memory and / or non-volatile computer-readable storage media. Further details regarding the memory 1040 are provided below.

[0102] The storage device 1050 includes a computer-readable storage medium. The storage device 1050 stores a larger amount of information and stores it for a longer time than the memory 1040. For example, the storage device 1050 includes storage media such as hard disks, optical discs, and solid-state drives.

[0103] The input device 1060 receives input from a user through tactile, video, audio, or touch input. For example, the input device 1060 includes a keyboard, a mouse, a touch screen, a microphone, or any other device that detects input from the user and sends the detected input to the recognition device 1020.

[0104] The output device 1070 provides the output of the recognition device 1020 to the user through visual, auditory, or tactile channels. The output device 1070 includes, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides output to the user.

[0105] The communication device 1080 communicates with external devices through a wired or wireless network. For example, the communication device 1080 receives the processed result data of image processing from the sensor device 900 through MIPI.

[0106] The devices, units, modules, apparatuses, and other components described herein are implemented by hardware components. Examples of hardware components suitable for performing the operations described in this application appropriately include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (such as, logic gate arrays, controllers, and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve the desired result). In one example, the processor or computer includes or is connected to one or more memories that store instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as, an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For the sake of brevity, the singular terms "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, can implement a single hardware component or two or more hardware components. The hardware components can have any one or more of different processing configurations, where examples of different processing configurations include: single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing. One or more processors can include hardware, such as, processors, controllers, and arithmetic logic units (ALUs), DSPs, microcomputers, FPGAs, programmable logic units (PLUs), microprocessors, or any other device capable of responding and executing instructions in a defined manner.

[0107] The method of performing the operations described in this application is executed by computing hardware (e.g., by one or more processors or computers), where the computing hardware is implemented as described above to execute instructions or software to perform the operations performed by the method described in this application. For example, a single operation or two or more operations can be executed by a single processor or two or more processors, or a processor and a controller. One or more operations can be executed by one or more processors, or a processor and a controller, and one or more other operations can be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, can execute a single operation or two or more operations.

[0108] The instructions or software for controlling a processor or computer to implement the hardware components and execute the method as described above are written as a computer program, code segment, instruction, or any combination thereof to individually or jointly direct or configure the processor or computer to operate as a machine or a special-purpose computer to perform the operations performed by the hardware components and the method as described above. In one example, the instructions or software include at least one of an applet, a dynamic link library (DLL), middleware, firmware, a device driver, and an application that stores an image data processing method. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executable by a processor or a computer. In another example, the instructions or software include high-level code executed by a processor or a computer using an interpreter. An ordinary programmer in the art can easily write the instructions or software based on the block diagrams and flowcharts shown in the drawings and the corresponding descriptions in the specification, and the block diagrams and flowcharts shown in the drawings and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware components and the method as described above.

[0109] Instructions or software for controlling computing hardware (e.g., one or more processors or calculators) to implement hardware components and execute the methods described above, along with any associated data, data files, and data structures, can be recorded, stored, or fixed in one or more non-transitory computer-readable storage media, or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk memory, hard disk drive (HDD), solid state drive (SSD), card memory (such as, multimedia card, secure digital (SD) card, or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to a processor or computer such that the processor or computer can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0110] Although this disclosure includes specific examples, it will be apparent after understanding the disclosure of this application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein will be considered only in a descriptive sense and not for purposes of limitation. The description of a feature or aspect in each example will be considered applicable to similar features or aspects in other examples. Appropriate results can be obtained if the described techniques are performed in a different order and / or if the components in the described system, architecture, device, or circuit are combined in a different manner and / or replaced or supplemented with other components or their equivalents. Accordingly, the scope of this disclosure is not limited by the specific embodiments, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents will be construed as being included in this disclosure.

Claims

1. A sensor device, comprising: an image sensor configured to acquire image data; an image buffer configured to store the image data; and an image processor configured to: generate processed image data by applying a filter corresponding to a storage mode of the image buffer to the image data stored in the image buffer, wherein the image buffer is configured to: sequentially store portions of the image data acquired by the image sensor in units of rows, and sequentially overwrite the rows of the image data, wherein the storage mode of the image buffer indicates a storage state of the image buffer, and the storage mode changes according to a row of the most recently stored image data among a plurality of rows included in a buffer area of the image buffer, wherein the number of filters of different modes corresponds to the number of rows of image data that can be stored in the image buffer, wherein each time image data is input to the image buffer, components of the filter are rotated to correspond to the order of the rows of the image data stored in the image buffer.

2. The sensor device according to claim 1, wherein, The image processor is further configured to: select a filter corresponding to the storage mode of the image buffer from a plurality of filters of different modes, and apply the selected filter to the image data stored in the image buffer.

3. The sensor device according to claim 2, wherein, The number of filters of different modes is determined based on the number of rows of image data that can be stored in the image buffer and the size of the filter.

4. The sensor device according to any one of claims 1 to 3, wherein, The image processor is further configured to: select a filter for the image data currently stored in the image buffer from a plurality of filters of a plurality of modes based on a buffer area of the most recently stored image data in the image buffer.

5. The sensor device according to any one of claims 1 to 3, wherein, The image processor is further configured to: generate image data of convolution processing as the processed image data by applying a convolution filter for convolution processing to the image data stored in the image buffer.

6. The sensor device according to claim 5, wherein, Convolution processing for the image data stored in the image buffer is performed in the sensor device.

7. The sensor device according to any one of claims 1 to 3, wherein, The image processor is further configured to: output a row of a result image by applying a convolution filter to the image data stored in the image buffer in response to a row of the image data being input and stored in the image buffer.

8. The sensor device according to any one of claims 1 to 3, wherein, The image buffer is further configured to: store image data of pixels including a plurality of color components arranged in a pattern, and The image processor is further configured to: generate processed image data by applying a filter corresponding to the pattern to the image data stored in the image buffer.

9. The sensor device according to claim 8, wherein, The image processor is further configured to: apply a first filter corresponding to an arrangement pattern of a first color component to the image data stored in the image buffer, and apply a second filter corresponding to an arrangement pattern of a second color component to the image data stored in the image buffer.

10. The sensor device according to claim 9, wherein, The first color component and the second color component are different color components, and The first filter and the second filter are alternately applied to pixels of the image data stored in the image buffer.

11. The sensor device according to any one of claims 1 to 3, wherein, The image processor is further configured to: apply a filter to the image data stored in the image buffer, and generate quantized image data by quantizing the image data to which the filter has been applied.

12. The sensor device according to any one of claims 1 to 3 further comprises: An output interface configured to send the image - processed data to the outside of the sensor device.

13. The sensor device according to claim 1, wherein, The image processor is further configured to: based on the most recently changed row of the image data in the image buffer, select a filter for the image data currently stored in the image buffer from a plurality of filters of a plurality of modes.

14. The sensor device according to claim 12, wherein, The output interface is further configured to: in response to an object being detected in the image - processed data, send the image - processed data to the outside of the sensor device.

15. An image data processing method, the image data processing method comprising: Obtaining image data; Storing the obtained image data in an image buffer; And Generating image - processed data by applying a filter corresponding to the storage mode of the image buffer to the image data stored in the image buffer, wherein the step of storing the obtained image data in the image buffer comprises: storing parts of the image data row - by - row sequentially and overwriting the rows of the image data sequentially, wherein the storage mode of the image buffer indicates the storage state of the image buffer, and the storage mode changes according to the row of the most recently stored image data among a plurality of rows included in the buffer area of the image buffer, wherein the number of filters of different modes corresponds to the number of rows of image data that can be stored in the image buffer wherein whenever image data is input into the image buffer, the components of the filter are rotated to correspond to the order of the rows of the image data stored in the image buffer.

16. The image data processing method according to claim 15, wherein, The step of generating image - processed data comprises: Selecting a filter corresponding to the storage mode of the image buffer from a plurality of filters of different modes and applying the selected filter to the image data stored in the image buffer.

17. The image data processing method according to claim 15, wherein, The step of generating image - processed data comprises: Generating image data for convolution processing as the image - processed data by applying a convolution filter for convolution processing to the image data stored in the image buffer.

18. The image data processing method according to claim 15, wherein, The step of generating image - processed data comprises: In response to a row of the obtained image data being input and stored in the image buffer, generating a row of a result image by applying a convolution filter to the image data stored in the image buffer.

19. The image data processing method according to claim 15, wherein, The image data includes pixels of a plurality of color components arranged in a pattern, and The step of generating image - processed data comprises: generating image - processed data by applying a filter corresponding to the pattern to the image data stored in the image buffer.

20. The image data processing method according to claim 19, wherein, The step of generating image - processed data comprises: Applying a first filter corresponding to the arrangement pattern of the first color component to the image data stored in the image buffer; and Applying a second filter corresponding to the arrangement pattern of the second color component to the image data stored in the image buffer.

21. A non - transitory computer - readable storage medium storing instructions, which when executed by a processor, cause the processor to execute the image data processing method according to any one of claims 15 to 20.

22. A sensor device, comprising: an image sensor configured to acquire image data; an image buffer configured to sequentially store portions of the image data in row units and sequentially overwrite rows of the image data; and an image processor configured to: process the stored image data by applying a filter corresponding to a storage pattern of a row of the image data most recently stored in the image buffer, wherein the storage pattern of the row of the image data indicates a storage state of the image buffer, and the storage pattern changes according to the row of the image data most recently stored among a plurality of rows included in a buffer area of the image buffer, wherein the number of filters of different patterns corresponds to the number of rows of the image data that can be stored in the image buffer, wherein whenever the image data is input to the image buffer, components of the filter are rotated to correspond to the order of the rows of the image data stored in the image buffer.

Images