Quantized image generation method and sensor debice for perfoming the same

Abstract

A method for generating a quantized image and a sensor device for performing the method for generating a quantized image are disclosed. The sensor device may include an image sensor for acquiring an image, an image processor for generating a quantized image by quantizing the acquired image using a learned quantization filter, and an output interface for transmitting the quantized image to the outside of the sensor device.

Description

Technology Field

[0001] The following embodiments relate to a technique for generating quantized images. Background Technology

[0003] A 'neural network' is a model that utilizes mathematical expressions to represent the characteristics of human biological nerve cells. Neural networks possess a computational structure in which a large number of processing elements with simple functions are connected in parallel, and they are widely used in various fields as a new technique to solve problems that were difficult to address with existing methodologies. To solve the problem of classifying input patterns into specific groups, neural networks employ algorithms that mimic the human ability to learn. Based on learned results, neural networks possess the ability to generate relatively correct outputs for input patterns that were not used in the learning process.

[0004] However, neural networks have a limitation in that they require significant resources for data processing. To overcome this limitation, research is currently being conducted on methods to reduce the bit width of neural networks while minimizing performance degradation. means of solving the problem

[0006] A sensor device according to one embodiment may include: an image sensor for acquiring an image; an image processor for generating a quantized image by quantizing the acquired image using a learned quantization filter; and an output interface for transmitting the quantized image to the outside of the sensor device.

[0007] The above quantized image may be an image with fewer bits than the image acquired by the image sensor.

[0008] The above-mentioned quantized image may be a binary image in which the pixel value of each pixel has either a first pixel value or a second pixel value.

[0009] The above quantization filter may be a dither matrix that includes multiple threshold values ​​determined through a learning process as elements.

[0010] The above dither matrix may be a multi-channel dither matrix that includes different threshold values ​​as elements for each channel.

[0011] The above learning process may include: a process of generating a quantized learning image by quantizing a learning image using the quantization filter; a process of obtaining a result label output from the object recognition model when the quantized learning image is input to the object recognition model; and a process of adjusting the elements of the quantization filter so as to reduce the loss based on the difference between the result label and the correct label.

[0012] The above learning process may include: a process of obtaining a first result label output from an object recognition model when a learning image is input to an object recognition model; a process of obtaining a second result label output from an object recognition model when a quantized image generated by quantizing the learning image using the quantization filter is input to the object recognition model; and a process of adjusting the elements of the quantization filter so as to reduce the loss based on the difference between the first result label and the second result label.

[0013] The image processor can generate the quantized image by performing a convolution operation on the image acquired by the image sensor using a learned convolution filter and quantizing the image on which the convolution operation was performed using a learned quantization filter.

[0014] The above image processor may include at least one of a digital signal processor (DSP), an image signal processor (ISP), and a micro controller unit (MCU).

[0015] The above quantized image is transmitted to an object recognition device, and the object recognition device can recognize an object appearing in the quantized image using an object recognition model having a bit width corresponding to the bit width of the quantized image.

[0016] A method for generating a quantized image of a sensor device including an image sensor, an image processor, and an output interface according to one embodiment may include: a step of acquiring an image using the image sensor; a step of generating a quantized image by the image processor quantizing the acquired image using a learned quantization filter; and a step of transmitting the quantized image to the outside of the sensor device through the output interface.

[0017] The generating step may include: a step of performing a convolution operation on an image acquired by the image sensor using a learned convolution filter; and a step of generating the quantized image by quantizing the image on which the convolution operation was performed using a learned quantization filter. Brief explanation of the drawing

[0019] FIG. 1 is a diagram illustrating the overall configuration of an object recognition system according to one embodiment. FIG. 2 is a diagram illustrating an object recognition process according to one embodiment. FIG. 3 is a flowchart illustrating the operation of a quantized image generation method according to one embodiment. FIGS. 4a and FIGS. 4b are drawings for explaining the process of generating a quantized image using a quantization filter according to one embodiment. FIGS. 5A and FIGS. 5B are drawings for explaining the learning process of a quantization filter according to one embodiment. FIG. 6 is a diagram illustrating an object recognition process according to another embodiment. FIG. 7 is a diagram illustrating the learning process of a convolution filter and a quantization filter according to one embodiment. FIG. 8 is a flowchart illustrating the operation of a quantized image generation method according to another embodiment. FIG. 9 is a diagram illustrating the configuration of a sensor device according to one embodiment. FIG. 10 is a diagram illustrating the configuration of an object recognition device according to one embodiment. Specific details for implementing the invention

[0020] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosed forms, and the scope of this specification includes modifications, equivalents, or substitutions that fall within the technical concept.

[0021] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. Furthermore, when a component is referred to as being "connected" to another component, it should be understood that it may be directly connected or coupled to that other component, or that there may be other components present in between.

[0022] Unless otherwise specified, the singular form should be interpreted to mean "one or more." In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0023] Furthermore, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0024] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0026] FIG. 1 is a diagram illustrating the overall configuration of an object recognition system according to one embodiment.

[0027] Referring to FIG. 1, an object recognition system (100) includes a sensor device (110) and an object recognition device (120). The object recognition system (100) is a system that acquires an image through the sensor device (110) and recognizes an object appearing in the image through the object recognition device (120). The object recognition system (100) can be used for face recognition or object recognition, etc.

[0028] The sensor device (110) acquires an image using an image sensor such as a camera and quantizes the acquired image to generate a quantized image. The quantized image is an image with fewer bits than the image acquired by the image sensor, and may be a binary image in which, for example, each pixel value of the entire image has one of a first pixel value (e.g., '0') and a second pixel value (e.g., '1').

[0029] In one embodiment, the sensor device (110) can quantize the image itself using a processing unit included internally, such as an image processor or a micro controller unit. The image processor can generate a quantized image by quantizing the image using a learned quantization filter. The quantized image can be transmitted to an object recognition device (120). In this way, the quantization processing of the image acquired at the sensor device (110) is performed, rather than the object recognition device (120) performing the processing of the image quantization. Since the quantization processing is performed at the sensor device (110) and the quantized image is transmitted from the sensor device (110), not only the amount of data transmitted but also the bandwidth required for data transmission can be reduced. In this specification, the image processor refers to a processor capable of performing the function of processing an image within the sensor device (110), and should not be interpreted as being limited to a processor that performs only the function of processing an image. According to the embodiment, the image processor may perform other functions (e.g., control of the sensor device (110), etc.) in addition to image processing of quantization processing.

[0030] A large amount of information may be lost during the process of quantizing an image, and such loss of information may affect the result of object recognition. The object recognition system (100) can reduce the degradation of object recognition performance caused by using a quantized image by minimizing important information lost during the image quantization process through the use of an optimal quantization filter learned through a learning process.

[0031] The object recognition device (120) receives an image quantized by the sensor device (110) from the sensor device (110) and performs object recognition based on the quantized image. The object recognition device (120) can perform object recognition on the quantized image using a learned object recognition model. The object recognition model may be a neural network model and may provide a score (e.g., expected value or probability value) regarding which object the object appearing in the quantized image corresponds to, based on image information (e.g., pixel value information) of the quantized image input to the object recognition model.

[0032] Neural network models used as object recognition models may be deep neural networks (DNNs), such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). However, the types of object recognition models are not limited to this, and neural network models such as autoencoders or generative adversarial networks (GANs) may also be used as object recognition models, and there are no restrictions on their types or forms.

[0033] In one embodiment, the object recognition model may be a neural network model having a bit width corresponding to the bit width of the quantized image. By using a neural network model having a low bit width according to the bit width of the quantized image, the amount of required resources can be reduced and the processing speed increased. In one embodiment, by using such a low bit width object recognition model, the object recognition system (100) can perform the object recognition process at high speed with low power consumption and low memory capacity usage in a limited embedded system such as a smart sensor or a smartphone.

[0035] FIG. 2 is a diagram illustrating an object recognition process according to one embodiment.

[0036] Referring to FIG. 2, the object recognition process may include a step (210) of generating a quantized image performed by a sensor device and a step (220) of performing object recognition based on the quantized image by an object recognition device.

[0037] In step (210), the sensor device acquires an image (212) using an image sensor and generates a quantized image (216) through an image quantization process (214). The image (212) acquired by the image sensor may be composed of pixel values ​​with relatively high bits, such as 8 bits, 10 bits, or 12 bits, for example, and the quantized image (216) may be composed of pixel values ​​with relatively low bits, such as 2 bits or 4 bits.

[0038] In the image quantization process (214), the sensor device can quantize the image (212) using a halftone technique. The halftone technique is a technique for converting an image with a high bit width into an image with a low bit width (e.g., a binary image), and is a technique for representing the pixel values ​​of an image as a density in 2D space. The sensor device can quantize the image (212) using, for example, a learnable quantization filter or by using an error spreading technique. An embodiment using an error spreading technique is described in more detail below with reference to FIG. 8. Below, the description focuses on an embodiment in which the image (212) is quantized using a quantization filter.

[0039] In one embodiment, the quantization filter used in the image quantization process (214) may be a dither matrix generated through a learning process. The dither matrix may have different threshold values ​​as elements for each location, and depending on the embodiment, the dither matrix may be composed of multiple channels. By using the dither matrix generated through the learning process, an optimized quantized image (216) can be generated as an input to an object recognition model (222) used for object recognition. The learning process of the dither matrix is ​​described in more detail below with reference to FIGS. 5a and 5b.

[0040] The quantized image (216) can be transmitted to an object recognition device through the output interface of the sensor device. In this way, the image (212) acquired by the image sensor is converted into a quantized image (216) having a smaller number of bits within the sensor device and transmitted to the object recognition device, thereby reducing the amount of data to be transmitted compared to the case where the acquired image (212) is transmitted directly to the object recognition device.

[0041] In step (220), the object recognition device may perform object recognition using an object recognition model (222). The object recognition model (222) may be, for example, a trained neural network model and may provide a recognition result for object recognition based on an input quantized image (216). The object recognition process may be performed under the control of a processor such as a CPU (central processing unit), GPU (graphic processing unit), or NPU (neural network processing unit) included in the object recognition device. In one embodiment, when the quantized image (216) is input to the object recognition model (222), the object recognition model (222) may output a score indicating the probability or possibility that an object appearing in the quantized image (216) corresponds to each class (or label). The object recognition device may determine an object recognition result based on information regarding the score.

[0042] Since the quantized image (216) consists of a small number of bits, it is suitable as an input to an object recognition model (222) with a low bit width. A low bit width refers to a small number of bits, such as 4 bits. A neural network model with a low bit width can be used as an object recognition model (222) for high-speed computation in a limited resource environment, such as low power and low-capacity memory. The quantized image (216), which is an image with a small number of bits, can be used as an input to the object recognition model (222). By using a learned quantization filter, the sensor device can reduce the degree of loss of important information during the image quantization process (214), and accordingly, the reliability and accuracy of the object recognition result can be increased.

[0044] FIG. 3 is a flowchart illustrating the operation of a quantized image generation method according to one embodiment. The quantized image generation method according to one embodiment can be performed by a sensor device.

[0045] Referring to FIG. 3, in step (310), the sensor device acquires an image using an image sensor such as a camera. The acquired image has a high bit count, and the sensor device performs image processing to reduce the bit count of the image through a processing unit, such as an internal image processor or a microcontroller unit. As part of this image processing process, in step (320), the sensor device generates a quantized image by quantizing the image acquired in step (310) using a learned quantization filter. The quantized image is an image with a lower bit count than the image acquired by the image sensor, and may be, for example, a binary image.

[0046] The quantization filter may be, for example, a dither matrix, and the sensor device can generate a quantized image by applying the dither matrix to an image acquired by an image sensor. The dither matrix may consist of one or more channels, and multiple threshold values ​​included as elements of the dither matrix may be determined through a learning process. Through the learning process, the elements of the dither matrix may be determined to minimize the degradation of object recognition performance that may occur when using the quantized image.

[0047] According to another embodiment, the sensor device may perform a convolution operation prior to the quantization process. For example, the sensor device may perform a convolution operation on an image acquired by an image sensor using a learned convolution filter. Subsequently, the sensor device may generate a quantized image by quantizing the image on which the convolution operation was performed using a learned quantization filter. The learning process for the convolution filter and the quantization filter is explained in more detail below with reference to FIG. 7.

[0048] In step (330), the sensor device can transmit the quantized image generated in step (330) to the outside of the sensor device through an output interface. As described above, the image acquired through the image sensor is quantized within the sensor device, and then the quantized image is output from the sensor device. The quantized image can be transmitted via wired or wireless connection through the output interface. In one embodiment, the image quantized by the sensor device is transmitted to an object recognition device, and the object recognition device can recognize an object appearing in the quantized image using an object recognition model having a bit width corresponding to the bit width of the quantized image.

[0050] FIGS. 4a and FIGS. 4b are drawings for explaining the process of generating a quantized image using a quantization filter according to one embodiment.

[0051] Referring to FIG. 4a, a sensor device according to one embodiment can generate a quantized image (440) by quantizing an image (410) using a learnable dither matrix (420) as a quantization filter. The dither matrix (420) may have different threshold values ​​as elements, and the size of the dither matrix (420) is not limited and can be set in various ways. According to an embodiment, as shown in FIG. 4b, the dither matrix (425) may be composed of multiple channels. For example, the dither matrix (425) may include three channels of dither matrices, and each channel may include different threshold values ​​as elements.

[0052] Returning to FIG. 4a, the sensor device can generate a quantized image (440) by applying a dither matrix (420) to the pixel values ​​of pixels included in the image (410) and then applying a quantization function (430) to the result of the application. For example, the sensor device can subtract a threshold value included in the dither matrix (420) from the pixel values ​​of the pixels to which the dither matrix (420) is applied among all the pixels of the image (410). For example, assuming that the pixel value of pixel A in the image (410) is '12' and the threshold value of the dither matrix (420) applied to pixel A is '5', the result of applying the dither matrix (420) includes assigning a value of '7' to pixel A. Subsequently, the sensor device can apply a quantization function (430) to the value of '7' of pixel A to replace the pixel value of pixel A with a quantized value. For example, if a Sign function is used as the quantization function (430), the pixel value of pixel A can be replaced with the quantized value of '1' because the value of '7' of pixel A is positive. If a value of '-2' is assigned to pixel B as a result of applying the dither matrix (420), the pixel value of pixel B can be replaced with the quantized value of '0' because the value of '-2' of pixel B is negative. The sensor device can perform this process for each pixel of the image (410) to generate a quantized image (440) in which the pixel value of each pixel has a quantized pixel value.

[0054] FIGS. 5A and FIGS. 5B are drawings for explaining the learning process of a quantization filter according to one embodiment.

[0055] The elements of the quantization filter used by the sensor device to generate a quantized image can be determined through a learning process. Below, we will explain the use of a dither matrix as an example of a quantization filter.

[0056] Referring to FIG. 5a, a training image (510) is first selected as training data, and a trainable quantization filter (520) is applied to the training image (510). The training image (510) may be, for example, a relatively high-bit image of 8 bits. In one embodiment, a quantization function (530) may be applied to the result of subtracting a threshold value included in the quantization filter (520) from the pixel values ​​of the pixels to which the quantization filter (520) is applied among all pixels of the training image (510). As an example of the quantization function (530), a Sign function may be used. The quantization function (530) may assign a quantized value to the pixel value of the corresponding pixel based on the above subtraction result, and a quantized training image (540) may be generated as a result of applying the quantization function (530). In this way, by quantizing the training image (510) using a quantization filter (520), a quantized training image (540), for example, a 1-bit binary image, can be generated.

[0057] The quantized training image (540) is input into an object recognition model (550), and when the quantized training image (540) is input into the object recognition model (550), a result label (560) output from the object recognition model (550) can be obtained. The object recognition model (550) may be based on a neural network model and may provide score information regarding which object the object appearing in the quantized training image (540) corresponds to, according to the calculation result of the said neural network model. A loss is calculated based on the difference between the result label (560) and the correct label (570) for the actual object appearing in the training image (510), and the elements of the quantization filter (520) (e.g., threshold values ​​of the dither matrix) may be adjusted according to a backpropagation algorithm so that the loss is reduced. The correct label (570) corresponds to a desired value corresponding to the training image (510).

[0058] As described above, the quantization filter (520) can be trained in a way that minimizes the degradation of object recognition performance caused by using the quantized image of the object recognition model (550). According to an embodiment, during the training process, not only the quantization filter (520) but also the object recognition model (550) may be trained, and the parameters of the object recognition model (550) may be adjusted.

[0059] The above learning process can be performed repeatedly for each of the various learning videos, and through many learning processes, the elements of the quantization filter (520) can be gradually updated to desirable values.

[0060] According to another embodiment, after an object recognition model (550) is trained based on a training image (510), a training process of a quantization filter (520) can be performed while keeping the parameters of the object recognition model (550) fixed. During the training process of the quantization filter (520), a first result label output from the object recognition model (550) can be obtained when the training image (510) is input to the object recognition model (550). Then, a second result label output from the object recognition model (550) can be obtained when a quantized image (530), generated by quantizing the training image (510) using the quantization filter (520) and the quantization function (530), is input to the object recognition model (550). Subsequently, a loss is calculated based on the difference between the first result label and the second result label, and the elements of the quantization filter (520) can be adjusted according to a backpropagation algorithm so that the corresponding loss is reduced. Here, the first result label and the second result label may take the form of, for example, an embedding vector.

[0061] FIG. 5b is a diagram illustrating an embodiment in which the quantization filter (525) is a dither matrix composed of multiple channels. The process of training the quantization filter (525) can be applied as described in FIG. 5a, except for adjusting the threshold values ​​of the dither matrix of each channel in the quantization filter (525) that forms multiple channels rather than a single-channel quantization filter (520), and a detailed explanation is omitted.

[0063] FIG. 6 is a drawing for explaining an embodiment different from the embodiment described with reference to FIG. 2.

[0064] Referring to FIG. 6, an object recognition process according to another embodiment may include a step (610) of generating a quantized image performed by a sensor device and a step (620) of an object recognition device performing object recognition based on the quantized image.

[0065] In step (610), the sensor device acquires an image (612) using an image sensor and performs a convolution operation by applying a convolution filter (614) to the acquired image (612). The convolution filter is a learnable convolution filter, and the filter characteristics can be determined through the learning process. The sensor device can generate a quantized image (618) by performing image quantization (616) on the image to which the convolution filter is applied. The sensor device can perform image quantization (616) using a learned quantization filter. By performing a convolution operation on the image (612) before image quantization (616) is performed, the image (612) can be adjusted so that image features related to the object are better revealed.

[0066] The quantized image (618) is transmitted to an object recognition device, and in step (620), the object recognition device can perform object recognition based on the quantized image (618). The object recognition device inputs the quantized image (618) into an object recognition model (622), and the object recognition model (622) can provide a recognition result for object recognition based on the quantized image (618). For example, the object recognition model (622) can provide information about a result label corresponding to the quantized image (618).

[0068] FIG. 7 is a diagram illustrating the learning process of a convolution filter and a quantization filter according to one embodiment.

[0069] Referring to FIG. 7, a training image (710) is provided as training data. The training image (710) may be, for example, a relatively high-bit image of 8 bits. Subsequently, a convolution operation (725) is performed on the training image (710) based on a trainable convolution filter (720), and image quantization (735) is performed on the training image on which the convolution operation was performed based on a trainable quantization filter (730). As a result of performing image quantization (735), a quantized training image (740) is generated, and the generated quantized training image (740) is input to an object recognition model (750). The object recognition model (750) outputs a result label (760) corresponding to the quantized training image (740), and a loss is defined based on the difference between the result label (760) and the correct label (770) corresponding to the original training image (710). Subsequently, the parameters or elements of the convolution filter (720) and the quantization filter (730) can be adjusted in a direction that minimizes the corresponding loss according to the backpropagation algorithm, etc. The above learning process can be performed repeatedly for each of the training images, and the convolution filter (720) and the quantization filter (730) can be gradually changed in a desirable way through many learning processes.

[0071] FIG. 8 is a flowchart illustrating the operation of a method for generating a quantized image using an error diffusion technique according to another embodiment. The error diffusion technique is a quantization method that compensates for a quantization error by propagating the quantization error caused by quantization to surrounding pixels. In one embodiment, when a sensor device quantizes an image using the error diffusion technique, it can reduce the range of the quantization error by using the midpoint of each quantization interval as the reference value of the quantization interval.

[0072] Referring to FIG. 8, in step (810), the sensor device acquires an image using an image sensor. In step (820), the sensor device selects a current pixel within the acquired image. For example, the current pixel may be selected according to the rasterscan order.

[0073] In step (830), it can be determined whether the pixel value of the selected current pixel is greater than a threshold value. For example, assuming the range of pixel values ​​of the pixels in the image is from 0 to 255, the threshold value can be the middle value, 127.5. However, the size of the threshold value is not limited to this.

[0074] If the pixel value of the current pixel is smaller than the threshold value, a first reference value is selected as a reference value for calculating the quantization error in step (840). Alternatively, if the pixel value of the current pixel is not smaller than the threshold value, a second reference value is selected as a reference value for calculating the quantization error in step (845). In one embodiment, the first reference value may correspond to 63.75, which is the midpoint of the first quantization interval (0 to 127.5), and the second reference value may correspond to 191.25, which is the midpoint of the second quantization interval (127.5 to 255).

[0075] In step (850), the sensor device can calculate an error corresponding to the current pixel based on the pixel value of the current pixel and the quantization interval containing the pixel value. For example, assuming the pixel value of the current pixel is 100, since the pixel value of the current pixel is smaller than the threshold value 127.5, a first reference value of 63.75 is selected in step (840), and 36.25, which is the first reference value of 63.75, can be calculated as the error by subtracting the pixel value of 100 from the pixel value of 63.75. If the pixel value of the current pixel is 140, since the current pixel value is larger than the threshold value 127.5, a second reference value of 191.25 is selected in step (840), and -51.25, which is the second reference value of 191.25, can be calculated as the error by subtracting the pixel value of 140 from the pixel value of 140.

[0076] In step (860), the sensor device may adjust the pixel values ​​of the surrounding pixels of the current pixel based on the error corresponding to the current pixel calculated in step (850). For example, the sensor device may add a value to the pixel values ​​of the surrounding pixels to which a specific weight has been applied to the error. The surrounding pixels may correspond to one or more pixels adjacent to the current pixel that have not yet undergone quantization processing, and the weights applied to each surrounding bipolar pixel may differ. In one embodiment, the sensor device may apply the error to the surrounding pixels according to the Floyd-Steinberg algorithm. In this case, the surrounding pixels may include pixels located to the right, below left, below, and below right, respectively, adjacent to the current pixel. In this way, the error corresponding to the current pixel is propagated to the surrounding pixels, the pixel values ​​of the surrounding pixels are adjusted, and image quantization is performed based on the adjusted pixel values.

[0077] In step (870), the sensor device can generate a quantized image by matching the pixel value of the current pixel to a quantized pixel value. In one embodiment, the sensor device may match the pixel value of the current pixel to a first quantized pixel value (e.g., 0) if the pixel value of the current pixel falls within a first quantization interval, and match the pixel value of the current pixel to a second quantized pixel value (e.g., 1) if the pixel value of the current pixel falls within a second quantization interval. According to the embodiment, the quantization of the image may proceed in binary, bipolar, or ternary, but the scope of the embodiment is not limited thereto. In the case of ternary, the pixel value of the image may be quantized by mapping it to any one of -1, 0, and 1, and there may be three quantization intervals. The reference value of each quantization interval may be set to the midpoint of each quantization interval.

[0078] As shown above, the quantization error can be minimized by setting the midpoint of the quantization interval as the reference value. For example, if the first reference value is 0 and the second reference value is 255, the quantization error ranges from -127 to 127; however, if the first reference value is set to 63.75 and the second reference value to 191.25 as shown above, the quantization error ranges from -63.75 to 63.75. Accordingly, the range of the quantization error is minimized, thereby reducing the error between the quantized image and the original image.

[0079] As a reference value for the quantization interval, an intermediate value of the quantization interval may be used as described above, or any value other than the minimum value (e.g., 0) and the maximum value (255) may be used. Depending on the embodiment, the reference value of the quantization interval may be determined through a learning process.

[0081] FIG. 9 is a diagram illustrating the configuration of a sensor device according to one embodiment.

[0082] Referring to FIG. 9, the sensor device (900) includes an image sensor (910), an image processor (920), a memory (930), and an output interface (940). The sensor device (900) may correspond to the sensor device described in this specification.

[0083] The image sensor (910) acquires an image. For example, the image sensor (910) can acquire an image such as a color image, a grayscale image, or an infrared image.

[0084] The image processor (920) can control the operation of the sensor device (900) and may include, for example, one or more of a digital signal processor (DSP), an image signal processor (ISP), and a microcontroller unit (MCU). The image processor (920) can generate a quantized image by performing image quantization processing on an image acquired by an image sensor (910). The image processor (920) can perform one or more operations related to the image quantization described above through FIGS. 1 to 8.

[0085] For example, the image processor (920) can generate a quantized image (e.g., a binary image) by quantizing the acquired image using a learned quantization filter (e.g., a dither matrix). As another example, the image processor (920) can generate a quantized image by performing a convolution operation on the image acquired by the image sensor (910) using a learned convolution filter, and quantizing the image on which the convolution operation was performed using a learned quantization filter. As yet another example, the image processor (920) can generate a quantized image using an error diffusion technique as described in FIG. 8. The image processor (920) can calculate an error corresponding to the current pixel based on the pixel value of the current pixel included in the image and the quantization interval containing the pixel value, adjust the pixel value of the surrounding pixels of the current pixel based on the calculated error, and generate a quantized image based on the adjustment result.

[0086] The memory (930) can store instructions to be executed by the processor (920) and information for performing image quantization. The memory (930) can store information about an image acquired by the image sensor (910) and a quantized image generated through image quantization processing. The memory (930) may include, for example, high-speed random access memory and / or a non-volatile computer-readable storage medium.

[0087] The output interface (940) can transmit the quantized image generated by the image processor (920) to the outside of the sensor device (900). For example, the output interface (940) can transmit the quantized image to an object recognition device via wired or wireless communication.

[0089] FIG. 10 is a diagram illustrating the configuration of an object recognition device according to one embodiment.

[0090] Referring to FIG. 10, the object recognition device (1020) may include a processor (1030), memory (1040), storage device (1050), input device (1060), output device (1070), and network interface. Each component of the object recognition device (1020) may communicate with each other via a communication bus. The object recognition device (1020) may correspond to the object recognition device described herein.

[0091] The processor (1030) controls the operation of the object recognition device (1020) and can execute functions and instructions for performing object recognition. For example, the processor (1030) can execute instructions stored in memory (1040) or storage device (1050). The processor (1030) may include, for example, one or more of a CPU, GPU, and NPU, and can perform one or more operations related to object recognition as described through FIGS. 1 to 9. For example, the processor (1030) can perform object recognition based on a quantized image received from a sensor device (900).

[0092] The memory (1040) can store instructions to be executed by the processor (1030) and information for performing object recognition. The memory (1040) may include, for example, high-speed random access memory and / or a non-volatile computer-readable storage medium.

[0093] The storage device (1050) includes a computer-readable storage medium. According to one embodiment, the storage device (1050) can store a larger amount of information than the memory (1040) and can store information for a long period of time. For example, the storage device (1050) may include a hard disk, an optical disk, a solid-state drive, or a flash memory. Information about an object recognition model used for object recognition may be stored in the memory (1040) or the storage device (1050).

[0094] The input device (1060) can receive input from a user through tactile, video, audio, or touch input. For example, the input device (1060) may include a keyboard, a mouse, a touch screen, a microphone, or any other device capable of detecting input from a user and transmitting the detected input to an electronic device.

[0095] The output device (1070) may provide the output of the electronic device to the user through a visual, auditory, or tactile channel. The output device (1070) may include, for example, a display, a touch screen, a speaker, a vibration generating device, or any other device capable of providing the output to the user.

[0096] The communication device (1080) can communicate with an external device via a wired or wireless network. For example, the communication device (1080) can receive a quantized image from the sensor device (900).

[0098] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0099] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0100] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0101] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0102] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below. Explanation of the symbols

[0104] 100: Object recognition system 110, 900, 1010: Sensor device 120, 1020: Object recognition device 910: Image sensor 920: Image processor 940: Output interface 1030: Processor 930, 1040: Memory 1050: Storage device 1060: Input device 1070: Output device 1080: Network interface

Claims

Claim 1 A sensor device comprising: an image sensor for acquiring an image; an image processor for generating a quantized image by quantizing the acquired image using a learned quantization filter; and an output interface for transmitting the quantized image to the outside of the sensor device, wherein the image processor determines a reference value based on a threshold value determined according to a range of pixel values ​​of a pixel in the acquired image and the pixel value of the pixel, determines an error between the reference value and the pixel value of the pixel in a quantization interval that includes the pixel value of the pixel, adjusts the pixel values ​​of surrounding pixels of the pixel based on the determined error, and generates the quantized image by matching the pixel value of the pixel to the quantized pixel value. Claim 2 A sensor device according to claim 1, wherein the quantized image is an image with fewer bits than the image acquired by the image sensor. Claim 3 A sensor device according to claim 1, wherein the quantized image is a binary image in which the pixel value of each pixel has a value of either a first pixel value or a second pixel value. Claim 4 In claim 1, the sensor device wherein the quantization filter is a dither matrix comprising a plurality of threshold values ​​determined through a learning process as elements. Claim 5 In paragraph 4, the sensor device wherein the dither matrix is ​​a multi-channel dither matrix comprising different threshold values ​​as elements for each channel. Claim 6 In claim 4, the sensor device comprises a learning process that generates a quantized learning image by quantizing a learning image using the quantization filter, and adjusts the elements of the quantization filter based on result labels output from the object recognition model when the quantized learning image is input to the object recognition model. Claim 7 In claim 6, the sensor device, wherein the learning process comprises a process of adjusting the elements of the quantization filter so as to reduce the loss based on the difference between the result label and the correct label. Claim 8 A sensor device according to claim 1, wherein the image processor performs a convolution operation on an image acquired by the image sensor using a learned convolution filter, and generates the quantized image by quantizing the image on which the convolution operation is performed using a learned quantization filter. Claim 9 A sensor device according to claim 1, wherein the image processor comprises at least one of a digital signal processor (DSP), an image signal processor (ISP), and a micro controller unit (MCU). Claim 10 A sensor device according to claim 1, wherein the quantized image is transmitted to an object recognition device, and the object recognition device recognizes an object appearing in the quantized image using an object recognition model having a bit width corresponding to the bit width of the quantized image. Claim 11 A method for generating a quantized image of a sensor device comprising an image sensor, an image processor, and an output interface, comprising: a step of acquiring an image using the image sensor; a step of generating a quantized image by the image processor quantizing the acquired image using a learned quantization filter; and a step of transmitting the quantized image to the outside of the sensor device through the output interface, wherein the step of generating the quantized image comprises: a step of determining a reference value based on a threshold value determined according to a range of pixel values ​​of a pixel in the acquired image and a pixel value of the pixel; a step of determining an error between the reference value of a quantization interval containing the pixel value of the pixel and the pixel value of the pixel; a step of adjusting the pixel values ​​of surrounding pixels of the pixel based on the determined error; and a step of matching the pixel value of the pixel to a quantized pixel value. Claim 12 A method for generating a quantized image according to claim 11, wherein the quantized image is an image with fewer bits than the image acquired by the image sensor. Claim 13 A method for generating a quantized image according to claim 11, wherein the quantization filter is a dither matrix comprising a plurality of threshold values ​​determined through a learning process as elements. Claim 14 A method for generating a quantized image according to claim 13, wherein the learning process comprises: a step of generating a quantized learning image by quantizing a learning image using the quantization filter; a step of obtaining a result label output from an object recognition model when the quantized learning image is input to an object recognition model; and a step of adjusting the elements of the quantization filter so as to reduce the loss based on the difference between the result label and the correct label. Claim 15 In claim 13, the learning process comprises: a step of obtaining a first result label output from an object recognition model when a learning image is input to an object recognition model; a step of obtaining a second result label output from an object recognition model when a quantized image generated by quantizing the learning image using the quantization filter is input to the object recognition model; and a step of adjusting the elements of the quantization filter so as to reduce the loss based on the difference between the first result label and the second result label. Claim 16 A method for generating a quantized image according to claim 11, wherein the generating step comprises: a step of performing a convolution operation on an image acquired by the image sensor using a learned convolution filter; and a step of generating the quantized image by quantizing the image on which the convolution operation was performed using a learned quantization filter. Claim 17 A method for generating a quantized image according to claim 13, wherein the dither matrix is ​​a multi-channel dither matrix that includes different threshold values ​​as elements for each channel. Claim 18 A method for generating a quantized image according to claim 11, wherein the quantized image is transmitted to an object recognition device, and the object recognition device recognizes an object appearing in the quantized image using an object recognition model having a bit width corresponding to the bit width of the quantized image. Claim 19 A computer-readable storage medium storing instructions for carrying out the method of any one of claims 11 through 18.

Images