System and method for processing images

Abstract

A system and method for processing an image are provided. A system includes: an image sensor configured to acquire an image; an image processor configured to generate a quantized image using a trained quantization filter based on the acquired image; and an output interface configured to output the quantized image.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0175530 filed on December 26, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following description relates to systems and methods for processing images. Background Art

[0003] The neural network may have an operation structure in which a large number of processing devices having simple functions are connected in parallel. In order to solve the problem of classifying input patterns into predetermined groups, the neural network may be implemented as an algorithm having a learning ability. The neural network may have a generalization ability to generate a relatively correct output for an input pattern that has not been used for training based on the result of training.

[0004] However, neural networks may have the limitation of requiring large amounts of resources to process data. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, a system includes an image sensor configured to acquire an image; an image processor configured to generate a quantized image using a trained quantization filter based on the acquired image; and an output interface configured to output the quantized image.

[0007] The quantized image may be an image having a smaller number of bits than that of an image acquired by an image sensor.

[0008] The quantized image may be a binary image including pixels each having one of a first pixel value and a second pixel value.

[0009] The quantization filter may be a dithering matrix and include, as elements, a plurality of threshold values ​​determined based on a training process.

[0010] The dithering matrix may include a plurality of channels and a different threshold value among the plurality of threshold values ​​for each of the plurality of channels as elements.

[0011] The training process may include: generating a quantized training image by quantizing the training image using a quantization filter; and determining the element by adjusting the element of the quantization filter based on a result label output from the object recognition model in response to the quantized training image being input to the object recognition model.

[0012] The step of adjusting the elements may include adjusting elements of a quantization filter to reduce a loss determined based on a difference between the result label and the correct answer label.

[0013] To generate the quantized image, the image processor may be configured to: perform a convolution operation on the acquired image using a trained convolution filter; and generate the quantized image by quantizing the image on which the convolution operation has been performed using the quantization filter.

[0014] The image processor may include any one or any combination of a digital signal processor (DSP), an image signal processor (ISP), and a microcontroller unit (MCU).

[0015] The system may include an object recognition device configured to recognize an object appearing in the quantized image using an object recognition model having a bit width corresponding to the bit width of the quantized image, and in order to output the quantized image, the output interface may be configured to transmit the quantized image to the object recognition device.

[0016] In another general aspect, a processor-implemented method includes acquiring an image using an image sensor; generating a quantized image using a trained quantization filter based on the acquired image using an image processor; and outputting the quantized image using an output interface.

[0017] The quantized image may be an image having a smaller number of bits than that of an image acquired by an image sensor.

[0018] The quantization filter may be a dithering matrix, and may include, as elements, a plurality of threshold values ​​determined based on a training process.

[0019] The dithering matrix may include a plurality of channels and a different threshold value among the plurality of threshold values ​​for each of the plurality of channels as elements.

[0020] The training process may include: generating a quantized training image by quantizing the training image using a quantization filter; obtaining a result label output from the object recognition model in response to the quantized training image being input to the object recognition model; and determining the element by adjusting an element of the quantization filter to reduce a loss determined based on a difference between the result label and a correct answer label.

[0021] The training process may include: obtaining a first result label output from an object recognition model in response to a training image being input to the object recognition model; obtaining a second result label output from the object recognition model in response to a quantized image generated by quantizing the training image using a quantization filter being input to the object recognition model; and determining the element by adjusting an element of the quantization filter to reduce a loss determined based on a difference between the first result label and the second result label.

[0022] The step of generating the quantized image may include: performing a convolution operation on the acquired image using a trained convolution filter; and generating the quantized image by quantizing the image on which the convolution operation is performed using the quantization filter.

[0023] Outputting the quantized image may include externally transmitting the quantized image to an object recognition device, and the object recognition device is configured to recognize an object appearing in the quantized image using an object recognition model having a bit width corresponding to a bit width of the quantized image.

[0024] A non-transitory computer-readable storage medium may store instructions that, when executed by a processor, configure the processor to perform the method.

[0025] In another general aspect, a processor-implemented method includes: acquiring an image; using an image processor, generating a quantized image by determining that a pixel value of the quantized image is any one of a first pixel value and a second pixel value based on comparing pixel values ​​of the acquired image with corresponding thresholds of a trained quantization filter; and outputting the quantized image using an output interface.

[0026] The threshold value may be a middle value of a range of pixel values ​​of the acquired image.

[0027] The step of determining the pixel value of the quantized image may include: in response to the pixel value of the acquired image being less than a corresponding threshold, determining the pixel value of the quantized image to be a first pixel value; and in response to the pixel value of the acquired image being greater than the corresponding threshold, determining the pixel value of the quantized image to be a second pixel value.

[0028] The method may include adjusting one or more pixel values ​​of the acquired image adjacent to the pixel value of the acquired image based on the determined error of the pixel value of the acquired image.

[0029] The method may include: in response to a pixel value of the acquired image being less than a threshold value, determining an error in the pixel value of the acquired image as a difference between the pixel value of the acquired image and a first reference value, the first reference value being a middle value of a range from a minimum pixel value of the acquired image to the threshold value.

[0030] The method may include: in response to a pixel value of the acquired image being greater than or equal to a threshold, determining an error in the pixel value of the acquired image as a difference between the pixel value of the acquired image and a second reference value, the second reference value being a middle value of a range from the threshold to a maximum pixel value of the acquired image.

[0031] The step of acquiring an image may include any one of: acquiring an image from an image sensor; or acquiring an image by performing a convolution operation on the image acquired by the image sensor.

[0032] In another general aspect, a processor-implemented method includes: generating a quantized training image by quantizing the training image using a quantization filter; obtaining a result label output from an object recognition model in response to the quantized training image being input to the object recognition model; and training the quantization filter by adjusting elements of the quantization filter to reduce a loss determined based on a difference between the result label and a correct answer label.

[0033] The method may include: acquiring an image using an image sensor; generating a quantized image using a trained quantization filter based on the acquired image using an image processor; and outputting the quantized image using an output interface.

[0034] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 An example of an object recognition system is shown.

[0036] Figure 2 An example of object recognition processing is shown.

[0037] Figure 3 An example of a quantized image generation method is shown.

[0038] Figure 4A and Figure 4B An example of a process of generating a quantized image using a quantization filter is shown.

[0039] Figure 5A and Figure 5B An example of a training process for a quantization filter is shown.

[0040] Figure 6 An example of object recognition processing is shown.

[0041] Figure 7 An example of a training process for convolution filters and quantization filters is shown.

[0042] Figure 8 An example of a quantized image generation method is shown.

[0043] Fig. 9 An example of a sensor device is shown.

[0044] Fig.10 An example of a device including a sensor device and an object recognition device is shown.

[0045] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features and structures. The drawings may not be to scale, and the relative sizes, proportions and depictions of the elements in the drawings may be exaggerated for clarity, illustration and convenience. DETAILED DESCRIPTION

[0046] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present application. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of known features may be omitted.

[0047] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein that will be clear after understanding the disclosure of the present application.

[0048] The specific structural or functional descriptions below describe examples only, and the scope of disclosure or such examples is not limited to the descriptions provided in this specification. Various substitutions and modifications may be made to the examples. Here, the examples are not to be construed as limited to the disclosure and should be understood to include all changes, equivalents, and substitutes within the disclosed concept and technical scope.

[0049] Although the terms "first" or "second" are used herein to describe various components, assemblies, regions, layers or parts, these components, assemblies, regions, layers or parts should not be limited by these terms. Instead, these terms are only used to distinguish one component, component, region, layer or part from another component, component, region, layer or part. Therefore, without departing from the teachings of the examples described herein, the first component, first component, first region, first layer or first part referred to in the examples may also be referred to as the second component, second component, second region, second layer or second part.

[0050] The terms used herein are only for the purpose of describing specific examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, as used herein, the singular form is also intended to include the plural form. As used herein, the term "and / or" includes any one of the relevant listed items and any combination of any two or more. As used herein, the terms "include", "comprise", "have" illustrate the existence of the narrated features, quantities, operations, elements, components and / or their combinations, but do not exclude the existence or addition of one or more other features, quantities, operations, elements, components and / or their combinations.

[0051] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those of ordinary skill in the art to which the present disclosure belongs and the same meaning as those commonly understood after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) will be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formalized meaning. The term "may" used herein for an example or embodiment (e.g., what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto.

[0052] Hereinafter, examples will be described in detail with reference to the accompanying drawings, and like reference numerals in the drawings denote like elements throughout.

[0053] Figure 1 An example of an object recognition system 100 is shown.

[0054] Reference Figure 1 , the object recognition system 100 may include a sensor device 110 and an object recognition device 120. The object recognition system 100 may be a system configured to acquire an image using the sensor device 110 and recognize an object appearing in the image using the object recognition device 120. For example, as a non-limiting example, the object recognition system 100 may be used for facial recognition or object recognition.

[0055] The sensor device 110 may acquire an image using an image sensor (e.g., a camera), and generate a quantized image by quantizing the acquired image. The quantized image may be an image having a smaller number of bits than the image acquired using the image sensor, and may include, for example, a binary image including pixels each having one of a first pixel value (e.g., "0") and a second pixel value (e.g., "1").

[0056] The sensor device 110 may automatically quantize the image using a processing unit (e.g., an image processor or a microcontroller unit (MCU)) included in the sensor device 110. The image processor may generate a quantized image by quantizing the image using a trained quantization filter. The quantized image may be transmitted to the object recognition device 120. As described above, the object recognition device 120 may perform processing of the quantized image acquired from the sensor device 110 instead of performing processing to quantize the image. The quantization processing may be performed in the sensor device 110, and the quantized image may be sent from the sensor device 110 to the object recognition device 120, and therefore, compared to when the acquired unquantized image is directly sent from the sensor device to the object recognition device, the object recognition system 100 of one or more embodiments may improve the technical fields of image quantization, image recognition, and computer functions by reducing both the amount of data and bandwidth used to send the image information from the sensor device 110 to the object recognition device 120. In the present disclosure, the image processor may be a processor configured to perform a function of processing an image in the sensor device 110, and should not be interpreted as being limited to a processor configured to perform only a function of processing an image. According to an example, the image processor may perform functions other than image processing of the quantization process (eg, control of the sensor device 110 ).

[0057] In a typical process of quantizing an image, a large amount of information may be lost. Such information loss may have an impact on the results of object recognition. The object recognition system 100 of one or more embodiments may improve the technical fields of image quantization, image recognition, and computer functions by minimizing the amount of important information that will be lost in the image quantization process by using an optimized quantization filter trained through a training process, thereby reducing the degradation of object recognition performance due to the use of quantized images.

[0058] The object recognition device 120 may receive the quantized image generated by the sensor device 110 from the sensor device 110 and perform object recognition based on the quantized image. The object recognition device 120 may perform object recognition on the quantized image using a trained object recognition model. The object recognition model may be, for example, a neural network model, and may provide a score (e.g., an expected value or a probability value) indicating the type of an object appearing in the quantized image based on image information (e.g., pixel value information) of the quantized image input to the object recognition model.

[0059] The neural network model used as the object recognition model may include, for example, a deep neural network (DNN) (e.g., a convolutional neural network (CNN) or a recurrent neural network (RNN)), however, the type of the object recognition model is not limited thereto. A neural network model (e.g., an autoencoder or a generative adversarial network (GAN)) may also be used as an object recognition model, and the type and form of the neural network are not limited thereto.

[0060] For example, the object recognition model can be a neural network model with a bit width corresponding to the bit width of the quantized image. By using a neural network model with a low bit width corresponding to the bit width of the quantized image, the object recognition model of one or more embodiments can improve the technical fields of image quantization, image recognition, and computer functions by reducing the amount of resources and increasing the processing speed for performing object recognition. In one example, the object recognition model of one or more embodiments can improve the technical fields of image quantization, image recognition, and computer functions by using a low bit width, so that the object recognition system 100 can perform object recognition processing with low power, low memory usage, and high speed in a limited embedded system (e.g., a smart sensor or a smart phone).

[0061] Figure 2 An example of object recognition processing is shown.

[0062] Reference Figure 2 , object recognition processing may include in a sensor device (e.g., Figure 1 The operation 210 of generating a quantized image in a sensor device 110) and the operation 210 of generating a quantized image in an object recognition device (e.g., Figure 1 In the object recognition device 120, an operation 220 of performing object recognition based on the quantized image is performed.

[0063] In operation 210, the sensor device may acquire an image 212 using an image sensor and generate a quantized image 216 through an image quantization process 214. The image 212 acquired using the image sensor may have pixel values ​​represented by a number of bits (e.g., 8 bits, 10 bits, or 12 bits) and the quantized image 216 may have pixel values ​​represented by a smaller number of bits (e.g., 2 bits or 4 bits).

[0064] In the image quantization process 214, the sensor device may quantize the image 212 using a halftone scheme. The halftone scheme may be a scheme for converting an image with a high bit width into an image with a low bit width (e.g., a binary image), and may also be a scheme for representing the pixel value of an image as a density in a two-dimensional (2D) space. The sensor device may quantize the image 212 using a trained quantization filter or using an error diffusion scheme. Figure 8 A non-limiting example of using an error diffusion scheme is further described below. In the following description, an example of quantizing an image 212 using a quantization filter is described.

[0065] The quantization filter used in the image quantization process 214 may be a jitter matrix generated through a training process. The jitter matrix may have different thresholds for each position as elements. According to an example, the jitter matrix may include a plurality of channels. By using the jitter matrix generated through the training process, an optimized quantized image 216 as an input of an object recognition model 222 for object recognition may be generated. Figure 5A and Figure 5B A non-limiting example of a training process for a dither matrix is ​​further described.

[0066] The quantized image 216 may be transmitted to the object recognition device via the output interface of the sensor device. As described above, the image 212 acquired using the image sensor may be converted into the quantized image 216 having a smaller number of bits than the number of bits of the image 212 in the image sensor, and the quantized image 216 may be transmitted to the object recognition device, thereby reducing the amount of data transmitted compared to when the acquired image is directly transmitted to the object recognition device.

[0067] In operation 220, the object recognition device may perform object recognition using an object recognition model 222. The object recognition model 222 may be, for example, a trained neural network model and may provide a recognition result of object recognition based on the input quantized image 216. The object recognition process may be performed under the control of a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processor (NPU)) included in the object recognition device. For example, when the quantized image 216 is input to the object recognition model 222, the object recognition model 222 may output a score indicating the likelihood or probability that an object appearing in the quantized image 216 corresponds to each category or label. In this example, the object recognition device may determine the object recognition result based on information associated with the score.

[0068] When the quantized image 216 has a small number of bits, the quantized image 216 can be input to an object recognition model 222 having a low bit width. The low bit width means, for example, a small number of bits (e.g., 4 bits). In a limited resource environment (e.g., low power and low capacity memory), a neural network model with a low bit width can be used as an object recognition model 222 for high-speed operation. The quantized image 216, which is an image with a small number of bits, can be used as an input to the object recognition model 222. The sensor device can use the trained quantization filter to reduce the degree of loss of important information in the image quantization process 214, so the sensor device of one or more embodiments can improve the accuracy and confidence of the object recognition result.

[0069] Figure 3 An example of a quantized image generation method is shown. Figure 3 The quantitative image generation method can be performed by, for example, a sensor device (e.g., Figure 1The sensor device 110) is used to perform.

[0070] Reference Figure 3 In operation 310, the sensor device may acquire an image using an image sensor (e.g., a camera). The acquired image may have a large number of bits, and the sensor device may automatically perform image processing to reduce the number of bits of the image using a processing unit (such as an image processor or an MCU) included in the sensor device. As in the above image processing process, in operation 320, the sensor device may generate a quantized image by quantizing the image acquired in operation 310 using a trained quantization filter. The quantized image may have a smaller number of bits than the image acquired using the image sensor, and may be, for example, a binary image.

[0071] The quantization filter may be, for example, a dithering matrix, and the sensor device may apply the dithering matrix to an image acquired using the image sensor to generate a quantized image. The dithering matrix may include a single channel or multiple channels, and multiple thresholds included as elements of the dithering matrix may be determined through a training process. Through the training process, the elements of the dithering matrix may be determined to minimize the degradation in object recognition performance that may occur due to the use of the quantized image.

[0072] For example, the sensor device may perform a convolution operation before the quantization process. The sensor device may perform a convolution operation on an image acquired using an image sensor using a trained convolution filter. The sensor device may generate a quantized image by quantizing the image on which the convolution operation is performed using the trained quantization filter. Figure 7 A non-limiting example of a training process for convolutional filters and quantization filters is further described.

[0073] In operation 330, the sensor device may transmit the quantized image generated in operation 320 to the outside of the sensor device via the output interface. As described above, the image acquired using the image sensor may be quantized in the sensor device and the quantized image may be output from the sensor device. The quantized image may be transmitted by wire or wirelessly using the output interface. For example, the sensor device may transmit the quantized image to an object recognition device, and the object recognition device may use an object recognition model having a bit width corresponding to the bit width of the quantized image to recognize an object appearing in the quantized image.

[0074] Figure 4A and Figure 4B An example of a process of generating a quantized image using a quantization filter is shown.

[0075] Reference Figure 4A , sensor devices (e.g., Figure 1The sensor device 110) can generate a quantized image 440 by quantizing the image 410 using the trained jitter matrix 420 as a quantization filter. The jitter matrix 420 may have different thresholds as elements, and the size of the jitter matrix 420 is not limited and may be variously set. Figure 4B As shown in , the dither matrix 425 may include multiple channels. For example, the dither matrix 425 may include a dither matrix for three channels, and may have different thresholds as elements for each channel. For example, the thresholds in a single channel of the multiple channels are different, and the thresholds may be the same or different between the multiple channels.

[0076] Return to reference Figure 4A , the sensor device may apply the dither matrix 420 to the pixel values ​​of the pixels included in the image 410, and apply the quantization function 430 to the results obtained by applying the dither matrix 420 to the pixel values ​​to generate the quantized image 440. For example, the sensor device may subtract the threshold value included in the dither matrix 420 from the pixel values ​​of the pixels to which the dither matrix 420 is to be applied among all the pixels in the image 410. In one example, when the pixel value of pixel A in the image 410 is "12" and the threshold value of the dither matrix 420 to be applied to the pixel A is "5", the value "7" may be assigned to the pixel A as the result obtained by applying the dither matrix 420. In this example, when generating the quantized image 440, the sensor device may apply the quantization function 430 to the value "7" of the pixel A to replace the pixel value of the pixel A with the quantized value. For example, when a sign function is used as the quantization function 430, in response to the value "7" of the pixel A being a positive value, the pixel value of the pixel A may be replaced with the quantized value "1". In another example, when the value "-2" is assigned to the pixel B as a result of applying the dither matrix 420, in response to the value "-2" of the pixel B being a negative value, the pixel value of the pixel B may be replaced with the quantized value "0" as a result of applying the quantization function 430. Therefore, in one example, when the corresponding pixel generated by applying the dither matrix 420 to the pixel of the image 410 has a positive value, the pixel of the quantized image 440 may be generated to have a value of "1", and when the corresponding pixel generated by applying the dither matrix 420 to the pixel of the image 410 has a negative value, the pixel of the quantized image 440 may be generated to have a value of "0". The sensor device may perform the above process on each pixel in the image 410 to generate the quantized image 440 having pixels each having a quantized pixel value.

[0077] Figure 5A and Figure 5B An example of a training process for a quantization filter is shown.

[0078] The training process may be used to determine the sensor device (e.g., Figure 1 The sensor device 110) generates an element of a quantization filter for a quantized image. In the following description, a jitter matrix may be used as a quantization filter.

[0079] Reference Figure 5A , a training image 510 may be selected as training data, and a trainable quantization filter 520 may be applied to the training image 510. The training image 510 may be, for example, an image having a relatively large number of bits (e.g., 8 bits). For example, a quantization function 530 may be applied to a result obtained by subtracting a threshold value included in the quantization filter 520 from the pixel value of each pixel to which the quantization filter 520 is to be applied among all pixels in the training image 510. For example, a sign function may be used as the quantization function 530. The quantization function 530 may assign a quantization value to the pixel value of the corresponding pixel based on the result obtained by subtracting the threshold value, and a training image 540 quantized based on the result of applying the quantization function 530 may be generated. As described above, a quantized training image 540 (e.g., a 1-bit binary image) may be generated by quantizing the training image 510 using the quantization filter 520.

[0080] The quantized training image 540 may be input to the object recognition model 550. In response to inputting the quantized training image 540 to the object recognition model 550, a result label 560 output from the object recognition model 550 may be obtained. The object recognition model 550 may be based on a neural network model, and may provide score information about the type of an object appearing in the quantized training image 540 based on a calculation result of the neural network model. A loss may be calculated based on a difference between the result label 560 and a correct answer label 570 of a real or predetermined object appearing in the training image 510, and an element of the quantized filter 520 (e.g., a threshold of a jitter matrix) may be adjusted based on a back propagation algorithm to reduce the loss. The correct answer label 570 may correspond to an expected value corresponding to the training image 510.

[0081] As described above, the quantization filter 520 may be trained to minimize the degradation of object recognition performance due to the use of the quantized image of the object recognition model 550. According to an example, the object recognition model 550 and the quantization filter 520 may be trained in a training process, and thus, the parameters of the object recognition model 550 may also be adjusted.

[0082] The above training process may be repeatedly performed for each of the various training images, and the elements of the quantization filter 520 may be iteratively updated to the desired value through a large number of training processes. For example, the elements of the quantization filter 520 may be iteratively updated using the training process until the calculated loss is less than or equal to a predetermined threshold.

[0083] In another example, when the object recognition model 550 is trained based on the training image 510 and when the parameters of the object recognition model 550 are fixed, the training process of the quantization filter 520 may be performed. In the training process of the quantization filter 520, when the training image 510 is input to the object recognition model 550, a first result label 560 output from the object recognition model 550 may be obtained. When the quantized training image 540 generated by quantizing the training image 510 using the quantization filter 520 and the quantization function 530 is input to the object recognition model 550, a second result label 560 output from the object recognition model 550 may be obtained. The quantization filter 520 may be trained by calculating a loss based on a difference between the first result label 560 and the second result label 560, and adjusting elements of the quantization filter 520 based on a back-propagation algorithm to reduce the loss. In this example, the first result label 560 and the second result label 560 may be in the form of, for example, an embedded vector. In one example, the trained quantization filter 520 may correspond to Figure 4A The jitter matrix 420 is shown in FIG.

[0084] Figure 5B 5 shows an example in which the quantization filter 525 may be a dither matrix including multiple channels. In addition to the fact that the threshold for each channel of the dither matrix can be adjusted in the quantization filter 525 forming the multiple channels (instead of or in addition to the quantization filter 520 for a single channel), the above reference Figure 5A The training process described above is also applicable to the process of training the quantization filter 525 to train the quantization filter 525. In one example, the trained quantization filter 525 may correspond to Figure 4B The dither matrix 425.

[0085] Figure 6 An example of object recognition processing is shown.

[0086] Reference Figure 6 , object recognition processing includes processing in sensor devices (e.g., Figure 1 Operation 610 of generating a quantized image in a sensor device 110 of the present invention and in an object recognition device (e.g., Figure 1 In the object recognition device 120, an operation 620 of performing object recognition based on the quantized image is performed.

[0087] In operation 610, the sensor device may acquire an image 612 using an image sensor, and perform a convolution operation by applying a convolution filter to the acquired image 612 in operation 614. The convolution filter may be a trained convolution filter, and the trained convolution filter may be determined by a training process. The sensor device may perform image quantization 616 on the image 612 to which the convolution filter is applied and generate a quantized image 618. The sensor device may perform image quantization 616 using the trained quantization filter. By performing a convolution operation on the image 612 before performing image quantization 616, the image 612 may be adjusted to more accurately represent image features associated with an object.

[0088] The quantized image 618 may be transmitted to an object recognition device. In operation 620, the object recognition device may perform object recognition based on the quantized image 618. The object recognition device may input the quantized image 618 into an object recognition model 622, and the object recognition model 622 may provide a recognition result of the object recognition based on the quantized image 618. For example, the object recognition model 622 may provide information associated with a result label corresponding to the quantized image 618.

[0089] Figure 7 An example of a training process for convolution filters and quantization filters is shown.

[0090] Reference Figure 7 , a training image 710 may be provided as training data. The training image 710 is, for example, an image having a relatively large number of bits (e.g., 8 bits). A convolution operation 725 may be performed on the training image 710 based on a trainable convolution filter 720, and an image quantization 735 may be performed on the training image on which the convolution operation 725 is performed based on a trainable quantization filter 730. A quantized training image 740 may be generated as a result obtained by performing the image quantization 735, and the generated quantized training image 740 may be input to an object recognition model 750. The object recognition model 750 may output a result label 760 corresponding to the quantized training image 740, and a loss may be determined based on a difference between the result label 760 and a correct answer label 770 corresponding to the original training image 710. For example, the convolution filter 720 and the quantization filter 730 may be trained by adjusting elements or parameters of the convolution filter 720 and the quantization filter 730 based on a back propagation algorithm to minimize the loss. The above training process may be repeatedly performed for each training image, and the convolution filter 720 and the quantization filter 730 may be iteratively changed as desired through a large number of training processes. For example, the elements or parameters of the convolution filter 720 and the quantization filter 730 may be iteratively updated using the training process until the determined loss is less than or equal to a predetermined threshold. In one example, the trained quantization filter 730 may correspond to Figure 4A The dither matrix 420 and Figure 4B Any one of the jitter matrices 425.

[0091] Figure 8 An example of a quantized image generation method (e.g., using an error diffusion scheme) is shown. The error diffusion scheme may be a scheme that propagates a quantization error caused by quantization to neighboring pixels and compensates for the quantization error. For example, when an image is quantized using an error diffusion scheme, a sensor device (e.g., Figure 1 The sensor device 110 may use the middle value of each quantization interval as a reference value of the quantization interval to reduce the range of the quantization error.

[0092] Reference Figure 8 In operation 810, the sensor device may acquire an image using an image sensor. In operation 820, the sensor device may select a current pixel from the acquired image. For example, the current pixel may be selected in a raster scan order.

[0093] In operation 830, it is determined whether the pixel value of the selected current pixel is greater than a threshold value. For example, when the pixel value of a pixel in the image is within a range of "0" to "255", the threshold value may be a middle value of the range (i.e., "127.5"). For example, when the image sensor is configured to generate an image having pixel values ​​within a range from a minimum pixel value (e.g., "0") to a maximum pixel value (e.g., "255"), the sensor device may determine the threshold value as an average value of the minimum pixel value and the maximum pixel value (e.g., "127.5"). However, the threshold value is not limited thereto.

[0094] When the pixel value of the current pixel is less than the threshold value, the first reference value may be selected as the reference value for calculating the quantization error in operation 840. When the pixel value of the current pixel is not less than the threshold value, the second reference value may be selected as the reference value for calculating the quantization error in operation 845. For example, the first reference value may correspond to "63.75" as the middle value of the first quantization interval "0" to "127.5", and the second reference value may correspond to "191.25" as the middle value of the second quantization interval "127.5" to "255". For example, when the image sensor is configured to generate an image having pixel values ​​ranging from a minimum pixel value (e.g., "0") to a maximum pixel value (e.g., "255") and the threshold value is determined as an average value of the minimum pixel value and the maximum pixel value (e.g., "127.5"), the sensor device may determine the first reference value as the average value of the minimum pixel value and the threshold value (e.g., "63.75"), and may determine the second reference value as the average value of the threshold value and the maximum pixel value (e.g., "191.25").

[0095] In operation 850, the sensor device may calculate an error corresponding to the current pixel based on the pixel value of the current pixel and the quantization interval including the pixel value of the current pixel. In one example, when the pixel value of the current pixel is "100", in response to the pixel value of the current pixel being less than "127.5" as a threshold value, the first reference value "63.75" may be selected in operation 840. In this example, in operation 850, "36.25" obtained by subtracting the first reference value "63.75" from the pixel value "100" may be calculated as the error. In another example, when the pixel value of the current pixel is "140", in response to the pixel value of the current pixel being greater than "127.5" as a threshold value, the second reference value "191.25" may be selected in operation 845. In this example, in operation 850, "-51.25" obtained by subtracting the second reference value "191.25" from the pixel value "140" may be calculated as the error.

[0096] In operation 860, the sensor device may adjust the pixel values ​​of the neighboring pixels of the current pixel based on the error corresponding to the current pixel calculated in operation 850. For example, the sensor device may add the value obtained by applying a predetermined weight to the error to the pixel value of the neighboring pixel. The neighboring pixel may correspond to at least one pixel among the pixels adjacent to the current pixel that has not yet been quantized, and different weights may be applied to the neighboring bipolar pixels. For example, the sensor device may apply errors to the neighboring pixels based on the Floyd-Steinberg algorithm. In this example, the neighboring pixels may include pixels located at each of the right side, lower left side, lower side, and lower right side of the current pixel. As described above, the error corresponding to the current pixel may be propagated to the neighboring pixels, the pixel values ​​of the neighboring pixels may be adjusted, and the image quantization may be performed based on the adjusted pixel values.

[0097] In operation 870, the sensor device may generate a quantized image by matching the pixel value of the current pixel with the quantized pixel value. In one example, when the pixel value of the current pixel is included in a first quantization interval (e.g., an interval from a minimum pixel value to a threshold value), the sensor device may match the pixel value of the current pixel with a first quantized pixel value (e.g., "0"). In another example, when the pixel value of the current pixel is included in a second quantization interval (e.g., an interval from a threshold value to a maximum pixel value), the sensor device may match the pixel value of the current pixel with a second quantized pixel value (e.g., "1"). According to the example, the quantization of the image may be performed by binary quantization, bipolar quantization, or ternary quantization, however, the example is not limited thereto. In the example of ternary quantization, the pixel value of the image may be mapped to any one or any combination of "-1", "0", and "1", and the image may be quantized. In addition, there may be three quantization intervals. The reference value of each of the three quantization intervals may be set to the middle value of each of the three quantization intervals.

[0098] As described above, the middle value of the quantization interval may be set as a reference value to minimize the quantization error. For example, when the first reference value is "0" and when the second reference value is "255", the quantization error may have a range of "-127" to "127". As described above, when the first reference value and the second reference value are set to "63.75" and "191.25", respectively, the quantization error may have a range of "-63.75" to "63.75". Therefore, the range of the quantization error can be minimized to reduce the error between the quantized image and the original image.

[0099] As described above, the middle value of the quantization interval may be used as the reference value of the quantization interval, or all values ​​except the minimum value (e.g., "0") and the maximum value (e.g., "255") may be used as the reference value of the quantization interval. According to an example, the reference value of the quantization interval may also be determined through a training process. In one embodiment, Figure 8 The method shown in can be performed by an image processor of a sensor device.

[0100] Fig. 9 An example of a sensor device (eg, sensor device 900 ) is shown.

[0101] Reference Fig. 9 , the sensor device 900 may include an image sensor 910, an image processor 920 (eg, one or more processors), a memory 930, and an output interface 940. The sensor device 900 may correspond to the sensor device described in the present disclosure (eg, Figure 1 sensor device 110).

[0102] The image sensor 910 may acquire an image. For example, the image sensor 910 may acquire an image (eg, a color image, a grayscale image, or an infrared image).

[0103] The image processor 920 may control the operation of the sensor device 900 and may include, for example, any one or any combination of a digital signal processor (DSP), an image signal processor (ISP), and an MCU. The image processor 920 may generate a quantized image by performing an image quantization process on an image acquired by the image sensor 910. The image processor 920 may perform the same processing as described above with reference to FIG. Figures 1 to 8 Describes one or more or all operations associated with image quantization.

[0104] In one example, the image processor 920 may generate a quantized image (e.g., a binary image) by quantizing the acquired image using a trained quantization filter (e.g., a dithering matrix). In another example, the image processor 920 may generate a quantized image by performing a convolution operation on the image acquired by the image sensor 910 using a trained convolution filter and quantizing the image on which the convolution operation is performed using the trained quantization filter. In yet another example, the image processor 920 may generate a quantized image by using the above reference. Figure 8 The image processor 920 may calculate an error corresponding to a current pixel included in the image based on the pixel value of the current pixel and the quantization interval including the pixel value, may adjust the pixel values ​​of neighboring pixels of the current pixel based on the calculated error, and may generate a quantized image based on the adjustment result.

[0105] The memory 930 may store instructions to be executed by the processor 920 and information for performing image quantization. The memory 930 may store information associated with an image acquired by the image sensor 910 and a quantized image generated by an image quantization process. The memory 930 may include, for example, a high-speed random access memory (RAM) and / or a non-volatile computer-readable storage medium.

[0106] The output interface 940 may transmit the quantized image generated by the image processor 920 to the outside of the sensor device 900. For example, the output interface 940 may transmit the quantized image to an object recognition device (eg, Figure 1 or the object recognition device 120 discussed below. Fig.10 object recognition device 1020).

[0107] Fig.10 An example of a device or system including a sensor device (eg, sensor device 900 ) and an object recognition device (eg, object recognition device 1020 ) is shown.

[0108] Reference Fig.10 , the object recognition device 1020 may include a processor 1030 (e.g., one or more processors), a memory 1040, a storage device 1050, an input device 1060, an output device 1070, and a communication device 1080. The components of the object recognition device 1020 may communicate with each other via a communication bus. The object recognition device 1020 may correspond to the object recognition device described in the present disclosure (e.g., Figure 1 object recognition device 120).

[0109] The processor 1030 may control the operation of the object recognition device 1020 and execute functions or instructions for performing object recognition. For example, the processor 1030 may execute instructions stored in the memory 1040 or the storage device 1050. The processor 1030 may include, for example, any one or any combination of a CPU, a GPU, and an NPU, and may execute the same functions as those described above. Figures 1 to 9 For example, processor 1030 may perform object recognition based on the quantized image received from sensor device 900.

[0110] The memory 1040 may store instructions to be executed by the processor 1030 and information for performing object recognition. The memory 1040 may include, for example, a high-speed RAM and / or a non-volatile computer-readable storage medium.

[0111] The storage device 1050 may include a non-transitory computer-readable storage medium. The storage device 1050 may store a larger amount of information than the amount of information stored by the memory 1040 for a relatively long period of time. For example, the storage device 1050 may include a hard disk, an optical disk, or a solid-state drive (SSD). Information about an object recognition model for object recognition may be stored in the memory 1040 and / or the storage device 1050.

[0112] The input device 1060 may receive input from the user through tactile input, video input, audio input, and / or touch input. For example, the input device 1060 may detect input from a keyboard, a mouse, a touch screen, a microphone, and / or a user, and may include any other device configured to transmit the detected input to the object recognition device 1020.

[0113] The output device 1070 may provide the output of the object recognition device 1020 to the user through a visual channel, an auditory channel, and / or a tactile channel. The output device 1070 may include, for example, a display, a touch screen, a speaker, a vibration generator, or other devices configured to provide output to the user.

[0114] The communication device 1080 may communicate with an external device via a wired or wireless network. For example, the communication device 1080 may receive a quantized image from the sensor apparatus 900 via a mobile industry processor interface (MIPI).

[0115] Object recognition system, sensor device, object recognition device, image sensor, image processor, memory, output interface, processor, storage device, input device, output device, communication device, object recognition system 100, sensor device 110, object recognition device 120, sensor device 900, image sensor 910, image processor 920, memory 930, output interface 940, object recognition device 1020, processor 1030, memory 1040, storage device 1050, input device 1060, output device 1070, communication device 1080, devices, units, modules, apparatuses and references herein Figures 1 to 10Other components described are implemented by hardware components or represent hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. Hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or processors or computers can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component, or two or more hardware components. Hardware components can have any one or more of different processing configurations, and examples of different processing configurations include: a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instructions single data (MISD) multiprocessing, and multiple instructions multiple data (MIMD) multiprocessing.

[0116] Perform the operations described in this application Figures 1 to 10The method shown in is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as the execution instructions or software described above to perform the operations performed by the method described in this application. For example, a single operation, or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation, or two or more operations.

[0117] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform methods as described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. Instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions used herein, which disclose algorithms for performing operations performed by the hardware components and methods as described above.

[0118] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), programmable random access read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card or micro card (for example, secure digital (SD) or extreme digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner, and provide instructions or software and any associated data, data files and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed by one or more processors or computers in a distributed manner.

[0119] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are considered to be descriptive only, not for limiting purposes. The description of the features or aspects in each example is considered to be applicable to similar features or aspects in other examples. If the described technology is performed in a different order, and / or if the components in the described system, architecture, device or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents, suitable results can be achieved. Therefore, the scope of the disclosure is not limited by a specific embodiment, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as included in the disclosure.

Claims

1. A system for processing an image, comprising: an image sensor configured to acquire an image; an image processor configured to generate a quantized image using a trained quantization filter based on the acquired image; and The output interface is configured to output a quantized image. wherein the quantization filter includes as elements a plurality of thresholds determined based on a training process, The training process includes: generating a quantized training image by quantizing the training image using a quantization filter; and The elements are determined by adjusting elements of the quantization filter based on result labels output from the object recognition model in response to the quantized training image being input to the object recognition model.

2. The system according to claim 1, wherein: The quantized image is an image having a smaller number of bits than that of an image acquired by an image sensor.

3. The system according to claim 1, wherein: The quantized image is a binary image including pixels each having one of a first pixel value and a second pixel value.

4. The system according to claim 1, wherein: The quantization filter is a dither matrix.

5. The system according to claim 4, wherein: The dither matrix includes: Multiple channels, where the dither matrix has different thresholds for each channel.

6. The system according to claim 1, wherein: The step of adjusting the elements includes adjusting the elements of the quantization filter to reduce a loss determined based on a difference between the result label and the correct answer label.

7. The system according to any one of claims 1 to 6, wherein: To generate the quantized images, the image processor was configured to: performing a convolution operation on the acquired image using the trained convolution filter; and A quantized image is generated by quantizing the image on which the convolution operation has been performed using a quantization filter.

8. The system according to any one of claims 1 to 6, wherein: The image processor includes any one or any combination of a digital signal processor, an image signal processor, and a microcontroller unit.

9. The system according to any one of claims 1 to 6, further comprising: an object recognition device configured to recognize an object appearing in the quantized image using an object recognition model having a bit width corresponding to the bit width of the quantized image, and The output interface is configured to output the quantized image to the object recognition device.

10. A method for processing an image, comprising: Acquire an image using an image sensor; generating a quantized image using a trained quantization filter based on the acquired image using an image processor; and Use the output interface to output the quantized image. wherein the quantization filter includes as elements a plurality of thresholds determined based on a training process, The training process includes: generating a quantized training image by quantizing the training image using a quantization filter; and The elements are determined by adjusting elements of the quantization filter based on result labels output from the object recognition model in response to the quantized training image being input to the object recognition model.

11. The method according to claim 10, wherein: The quantized image is an image having a smaller number of bits than that of an image acquired by an image sensor.

12. The method according to claim 10, wherein: The quantization filter is a dither matrix.

13. The method according to claim 12, wherein: The dither matrix includes: Multiple channels, where the dither matrix has different thresholds for each channel.

14. The method according to claim 12, wherein: The step of determining the element comprises: The elements are determined by adjusting elements of a quantization filter to reduce a loss determined based on a difference between the result label and a correct answer label.

15. The method according to claim 12, wherein: The training process also includes: obtaining a first result label output from the object recognition model in response to the training image being input to the object recognition model; and The step of determining the element comprises: determining the element by adjusting an element of a quantization filter to reduce a loss determined based on a difference between the first result label and the result label.

16. The method according to any one of claims 10 to 15, wherein: The steps to generate the quantized image include: performing a convolution operation on the acquired image using the trained convolution filter; and A quantized image is generated by quantizing the image on which the convolution operation has been performed using a quantization filter.

17. The method according to any one of claims 10 to 15, wherein: The step of outputting the quantized image comprises: sending the quantized image to an object recognition device, and The object recognition device is configured to recognize an object appearing in the quantized image using an object recognition model having a bit width corresponding to a bit width of the quantized image.

18. A method for processing an image, comprising: generating a quantized training image by quantizing the training image using a quantization filter; obtaining a result label output from the object recognition model in response to the quantized training image being input to the object recognition model; and The quantization filter is trained for quantization of the training images by adjusting elements of the quantization filter to reduce a loss determined based on the difference between the result label and the correct answer label.

19. The method according to claim 18, further comprising: Acquire an image using an image sensor; generating a quantized image using a trained quantization filter based on the acquired image using an image processor; and Use the output interface to output the quantized image.

20. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, configure the processor to perform the method of any one of claims 10 to 19.

Images