Electronic device and method of operating the same

Abstract

The disclosure relates to an electronic device and an operating method thereof. The electronic device includes an image sensing device including an image sensor and an image processor. The image sensor is configured to sense light originating from an object and generate an image layer. The image processor is configured to generate a feature map by performing a convolution operation on the image layer and a fixed-size kernel, generate a final feature map by performing a convolution operation on a variable-size kernel and the feature map, and generate image data including a channel classified for each color with the final feature map.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2021-0109019, filed with the Korean Intellectual Property Office on August 18, 2021, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] This disclosure relates to an electronic device and its operating method, and more specifically to an electronic device and its operating method capable of converting a light source of an object into an image layer and converting the image layer into image data. Background Technology

[0004] With the development of content processing technologies and devices, images are being used more frequently in content creation. Images are a means of conveying information about an object to the user through vision, and they are more efficient than normal text or speech that relies solely on hearing. Images can be captured using image sensors such as charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS), and the image layers generated by the image sensors can be converted into image data by an image processor. Summary of the Invention

[0005] Embodiments of this disclosure provide an electronic device and a method of operating the same, which can reduce the computational complexity and shorten the computation time for generating image data by adjusting the size of the kernel and the receiving field according to the size of the object.

[0006] According to embodiments of this disclosure, an electronic device includes an image sensor and an image processor. The image sensor is configured to sense light originating from an object and generate an image layer. The image processor is configured to generate a feature map by performing a convolution operation on the image layer and a fixed-size kernel, to generate a final feature map by performing a convolution operation on a variable-size kernel and the feature map, and to generate image data including channels that classify the final feature map for each color.

[0007] According to embodiments of this disclosure, a method of operating an electronic device includes: converting an image of an object into an image layer; generating the image layer as an initial feature map by performing a convolution operation using a fixed-size kernel; and comparing the size of the object included in the initial feature map with the size of a receptive field. The method further includes: generating multiple feature maps by adjusting the size of a variable-size kernel and the receptive field based on the comparison result; and generating channels and generating feature data including the channels, wherein the final feature map in the multiple feature maps is categorized by the channels for each color.

[0008] According to this technology, images can be accurately detected based on the size of the object, and the complexity and computation time of operations used to convert image layers into image data can be improved. Attached Figure Description

[0009] Figure 1 This is a diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0010] Figure 2 This is a diagram illustrating an image sensor according to an embodiment of the present disclosure.

[0011] Figure 3 This is a diagram illustrating an image processor according to an embodiment of the present disclosure.

[0012] Figure 4 This is a diagram illustrating a fixed network according to an embodiment of the present disclosure.

[0013] Figure 5 This is a diagram illustrating a variable network according to an embodiment of the present disclosure.

[0014] Figure 6A , Figure 6B and Figure 6C This is a diagram illustrating the method for generating feature maps.

[0015] Figure 7A and Figure 7B This is a diagram illustrating a method for determining variable values ​​for component settings by size according to an embodiment of the present disclosure.

[0016] Figure 8A and Figure 8B This is a diagram illustrating how to change the acceptor domain when it has a minimum value.

[0017] Figure 9A and Figure 9B This is a diagram illustrating how to change the acceptor domain when it has a maximum value.

[0018] Figure 10A , Figure 10B and Figure 10C This is a diagram illustrating how to change the acceptor domain when it has intermediate values.

[0019] Figure 11 This is a diagram illustrating a method of operating an image detector according to an embodiment of the present disclosure. Detailed Implementation

[0020] The specific structural or functional descriptions of embodiments based on the concepts disclosed in this specification or application are for illustrative purposes only. Embodiments based on the concepts disclosed may be implemented in various forms and should not be construed as limited to the embodiments described in this specification or application.

[0021] Figure 1 This is a diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0022] refer to Figure 1 The electronic device 1000 can be configured to acquire an image of the object 100. The electronic device 1000 can be configured to store, display, or output image data DATA_IMG obtained through image processing to an external device. According to an embodiment, the electronic device 1000 can output the image data DATA_IMG to the host 110 according to a request received from the host 110.

[0023] In one embodiment, the electronic device 1000 can be implemented as a packaged module, component, etc. In this case, the electronic device 1000 can be mounted on the host 110. The host 110 can be implemented as various electronic devices. For example, the host 110 can be implemented as a digital camera, mobile device, smartphone, personal computer (PC), tablet PC, laptop, personal digital assistant (PDA), enterprise digital assistant (EDA), portable multimedia player (PMP), wearable device, black box, robot, autonomous vehicle, etc.

[0024] In another embodiment, the electronic device 1000 can be implemented as an electronic device separate from the host 110. For example, the electronic device 1000 can be an imaging device, digital camera, camcorder, closed-circuit television (CCTV), network camera, security camera, industrial vision camera, mobile device, smartphone, PC, tablet PC, laptop computer, personal digital assistant (PDA), enterprise digital assistant (EDA), portable multimedia player (PMP), wearable device, black box, robot, autonomous vehicle, in-vehicle vision camera, set-top box, game console, electronic dictionary, e-book reader, desktop computer, server, MP3 player, smart medical device, television, DVD player, speaker, refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, air purifier, smart mirror, smart window, electronic key, electronic picture frame, digital billboard, security control panel, etc. Here, the wearable device can be a smartwatch, ring, bracelet, anklet, necklace, glasses, contact lens, head-mounted device (HMD), skin pad, electronic tattoo, bio-implantable circuit, etc.

[0025] Electronic device 1000 can be configured to convert light source LS (i.e., light originating from object 100) reflected from object 100 into image data DATA_IMG. Here, object 100 can include any living organism or object to be photographed.

[0026] Electronic device 1000 can receive a light source LS for object 100, analyze the received light source LS, and output the light source as image data DATA_IMG. For example, electronic device 1000 may include image sensor 200 and image processor 300.

[0027] Image sensor 200 can be configured to convert light source LS into image layer IMG_LA as digital data. Image processor 300 can be configured to output image data DATA_IMG by compressing image layer IMG_LA. For example, image processor 300 can process image layer IMG_LA using deep learning techniques and generate image data DATA_IMG as the result of the operation.

[0028] Image data DATA_IMG output from image processor 300 can be transferred to storage device 2000 capable of storing the data. Although in Figure 1 The storage device 2000 is shown as an embodiment for storing images of object 100, but depending on the purpose of the electronic device 1000, the image data DATA_IMG can be transferred to a storage device different from the storage device 2000.

[0029] The image processor 300 according to an embodiment can be configured to analyze the size of the object 100 from the image layer IMG_LA, and determine the size of the receptive field based on the size of the object 100. Furthermore, the image processor 300 according to an embodiment can be configured to generate a feature map based on the size-adjusted receptive field, and output image data DATA_IMG based on the feature map.

[0030] Figure 2 This is a diagram illustrating an image sensor according to an embodiment of the present disclosure.

[0031] Reference Figure 2 The image sensor 200 can be implemented as a charge-coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) image sensor.

[0032] CCD image sensors offer less noise and better image quality than CMOS image sensors. However, CMOS image sensors are easier to drive and can be implemented using various scanning methods. Furthermore, because CMOS image sensors can integrate signal processing circuitry on a single chip, they are easier to miniaturize, and because CMOS image sensors are interchangeable with CMOS process technologies, manufacturing costs are lower. Because CMOS image sensors have low power consumption, they are easier to apply to mobile devices.

[0033] Despite Figure 2The illustration shows a CMOS image sensor as an example, but various image sensors capable of outputting a light source LS as an image layer IMG_LA (which is digital data) can be used. For example, image sensor 200 may include a first converter 210 and a second converter 220.

[0034] The first converter 210 can be configured to convert a light source LS into an analog image signal IMG_an. For example, the first converter 210 may include a plurality of pixels arranged in a two-dimensional matrix, and each pixel may output an analog image signal IMG_an corresponding to the light source LS. That is, each pixel may accumulate photocharge corresponding to the amount of light source LS received by a photodiode, and output the analog image signal IMG_an based on the current formed by the accumulated photocharge.

[0035] The second converter 220 can be configured to convert the analog image signal IMG_an into an image layer IMG_LA as a digital signal.

[0036] Figure 3 This is a diagram illustrating an image processor according to an embodiment of the present disclosure.

[0037] Reference Figure 3 The image processor 300 can be configured to convert the image layer IMG_LA into image data DATA_IMG and output the image data DATA_IMG. According to an embodiment, the image processor 300 includes a fixed network 310, a variable network 320, an image detector 330, and a data output circuit 340.

[0038] The fixed network 310 can be configured to receive an image layer IMG_LA corresponding to the first input layer for the object, process the image layer IMG_LA using a kernel of fixed size (also called a fixed-size kernel), and generate an initial feature map iFM. For example, the fixed network 310 can generate an initial feature map iFM with image depth by performing multiple convolution operations and pooling using filters with fixed kernel sizes.

[0039] Convolution can be an operation that performs matrix operations on the input image layer IMG_LA and the kernel. Pooling can be an operation that generates a feature map by extracting specific values ​​from the convolutional layer produced by the convolution operation. For example, pooling can be performed by extracting the maximum value or the average value from the convolutional layer, and various other methods can be used to perform pooling.

[0040] In some comparative cases, complex convolution operations are performed multiple times in the image detector because feature maps, including those ultimately generated from the initial feature map, are used when generating feature data in the image detector. Therefore, even if the object size is large, many convolution operations are performed to extract the feature map because fixed pooling is difficult to effectively reduce the feature map size. In this embodiment, since the initial feature map iFM can be used as data for the convolution operations of the variable network 320, and the variable network 320 can variably adjust the convolution and pooling sizes used to extract the feature map, fewer convolution operations can be performed than in the comparative cases. Furthermore, since a size suitable for the receptive field of the object can be used, the accuracy of the feature data DATA_FC can be increased.

[0041] Convolution operation refers to the learning algorithm or operation method using convolutional neural networks (CNN), which is a deep learning technique.

[0042] The variable network 320 can be configured to generate a Kth feature map (KFM) by performing convolution operations. For example, the variable network 320 can receive an initial feature map (iFM) as an input layer, adjust the receptive region based on the object size according to the input initial feature map (iFM), and perform convolution operations and pooling. Therefore, multiple feature maps generated in the variable network 320 can be generated based on the feature maps generated in the previous step. For example, the size of the receptive region used in the next convolution operation can be adjusted based on the feature maps generated in the previous step, and a convolution operation using the receptive region with the adjusted size can be performed to generate the next feature map. The final feature map generated from the feature maps generated in the variable network 320 can be output as the Kth feature map (KFM). In the variable network 320, since the size of the receptive region is adjusted according to the object size, unnecessary convolution operations can be omitted.

[0043] Image detector 330 can be configured to generate channels by classifying the Kth feature map KFM ultimately generated in variable network 320 for each color, and to generate feature data DATA_FC including the generated channels. In this embodiment, the feature map generated before the final generated Kth feature map is not used in image detector 330. Therefore, the convolutional operations required by the image detector can be reduced.

[0044] The data output circuit 340 can convert the detection result into data and output the data. For example, the data output circuit 340 can be configured to output feature data DATA_FC as is, or to output image data DATA_IMG by processing, compressing, or encrypting the feature data DATA_FC.

[0045] Figure 4 This is a diagram illustrating a fixed network according to an embodiment of the present disclosure.

[0046] Reference Figure 4 The fixed network 310 can be configured to generate an initial feature map iFM by performing convolutional operations and pooling with a fixed kernel size. For example, the fixed network 310 can be driven according to the operational methods of a Very Deep Convolutional Network (VGGNet). VGGNet can be configured with at least one convolutional layer and a pooling layer. The convolutional layer can be the result of a convolutional operation between the image layer IMG_LA corresponding to the initial input layer for the object and a kernel, while the pooling layer can be the result of extracting specific values ​​from the convolutional layer. Therefore, the pooling layer of the fixed network 310 can be a feature map of the image layer IMG_LA. For example, multiple convolutional layers can be generated relative to the image layer IMG_LA, and a single pooling layer can be generated for the multiple convolutional layers. As the number of convolutional and pooling layers increases, the depth of the image can increase.

[0047] Assuming Figure 4 As shown, VGGNet can generate layer groups LG1 to LG5. In this embodiment, VGGNet can be configured to generate some of these layer groups. For example, in this embodiment, VGGNet can be configured to generate only layer groups LG1 to LG4, and not generate layer group LG5. Since the depth of the image is proportional to the number of layer groups, more accurate feature maps can be generated as the number of layer groups increases. However, in this embodiment, since the feature maps generated in the fixed network 310 are not in... Figure 3 Instead of being used directly in the image detector 330, it is only used for Figure 3 The variable network 320 allows for the generation of initial feature maps (iFM) using only some layer groups instead of all layer groups. In this embodiment, the first layer group LG1 to the fourth layer group LG4 are generated. However, VGGNet can be configured to generate all layer groups that can be generated by VGGNet, and it can also be configured to generate fewer layer groups than the first layer group LG1 to the fourth layer group LG4. As mentioned above, when only some layer groups are generated in VGGNet, the computation time required in VGGNet can be shortened, and the computational complexity can also be reduced.

[0048] The first layer group LG1 may include a first convolutional layer Conv1-1 and a first convolutional layer Conv1-2 generated by convolutional operations on the image layer IMG_LA, and a first pooling layer PL1 generated by processing the first convolutional layer Conv1-1 and the first convolutional layer Conv1-2. The second layer group LG2 may include a second convolutional layer Conv2-1 and a second convolutional layer Conv2-2 generated by convolutional operations on the first pooling layer PL1, and a second pooling layer PL2 generated by processing the second convolutional layer Conv2-1 and the second convolutional layer Conv2-2. The third layer group LG3 may include a third convolutional layer Conv3-1 to a third convolutional layer Conv3-3 generated by convolutional operations on the second pooling layer PL2, and a third pooling layer PL3 generated by processing the third convolutional layer Conv3-1 to the third convolutional layer Conv3-3. The fourth layer group LG4 may include convolutional layers Conv4-1 to Conv4-3 (4-1st convolutional layer) generated by convolutional operations on the third pooling layer PL3, and a fourth pooling layer PL4 generated by processing the convolutional layers Conv4-1 to Conv4-3. The fifth layer group LG5 may further include convolutional layers Conv5-1 to Conv5-3 (5-1st convolutional layer) generated by convolutional operations on the fourth pooling layer PL4, and a fifth pooling layer PL5 generated by processing the convolutional layers Conv5-1 to Conv5-3. As described above, in this embodiment, the operation of selecting layers included in the first layer groups LG1 to the fourth layer group LG4 and generating the fifth layer group LG5 can be omitted. However, all layers Conv1-1 to PL3 included in the first layer groups LG1 to the third layer groups LG3 and some layers Conv4-1 to Conv4-3 included in the fourth layer group LG4 can be selected. The pooling layer that is finally generated in VGGNet by performing processing operations on the selected layer can be output as the initial feature map iFM.

[0049] Figure 5 This is a diagram illustrating a variable network according to an embodiment of the present disclosure.

[0050] Reference Figure 5 The variable network 320 can be configured to use deep learning techniques in Figure 4 The initial feature map iFM generated in the fixed network 310 is used to generate the Kth feature map KFM. In the variable network 320, the size of the kernel (also called a variable-size kernel) can be adjusted according to the size of the image, and the size of the receptive field can also be adjusted by adjusting the size of the kernel. For example, the variable network 320 may include a first variable layer group VG1 to a Kth variable layer group VGK.

[0051] Each of the first variable layer group VG1 to the Kth variable layer group VGK can determine the kernel size based on the feature map generated in the previous layer group, and use the determined kernel size to perform convolution operations to generate feature maps. For example, when all images of objects included in a previous receptive region are included in that previous receptive region, the kernel size used in the next operation can remain the same as before, and therefore the size of the next receptive region can also remain the same as before. When the image of the object included in the receptive region is larger than the previous receptive region, the kernel size to be used in the next operation can be increased, and therefore the size of the next receptive region can also be increased.

[0052] In other words, since the sizes of the kernel and receptive field are adjusted according to the image size of the object, the final generated feature map is generated using a kernel and receptive field optimized for the image of the object. Therefore, in this embodiment, the Kth feature map KFM, which is the final feature map generated in the variable network 320, is used as... Figure 3 The input data of the image detector 330, and the feature maps generated before the Kth feature map KFM are not used as input data of the image detector 330.

[0053] The specific descriptions of the first variable layer group VG1 to the Kth variable layer group VGK are as follows.

[0054] The first variable layer VG1 can generate the first feature by processing the initial feature map iFM. Figure 1 FM, the second variable layer group VG2 can process the first feature Figure 1 FM to generate second features Figure 2 FM, and so on, until the final Kth variable layer group VGK can generate the Kth feature map KFM by processing the (K-1)th feature map (K-1)FM. When the Kth variable layer group VGK is the last group of variable layer groups included in the variable network 320, the Kth feature map KFM generated in the Kth variable layer group VGK is the final feature map.

[0055] The first variable layer group VG1 may include a first size determination component 1SDU and a first variable convolutional layer 1VCL. The first size determination component 1SDU may be configured to use data included in the initial feature map iFM to compare the image size of the object being processed with the size of a preset receptive field, and output a first compensation value 1VAL as the comparison result. For example, if the image size of the object is less than or equal to the size of the preset receptive field, and if the image size of the object is greater than the size of the preset receptive field, the first size determination component 1SDU may output a first compensation value 1VAL with different values. The first variable convolutional layer 1VCL may adjust the kernel size according to the first compensation value 1VAL, use the adjusted kernel as a filter to operate on the initial feature map iFM, and generate a first feature. Figure 1 FM is the result of the operation. For example, the first variable convolutional layer 1VCL can set a first receptive field with a first size according to the adjusted kernel size, and perform convolution operation on the initial feature map iFM according to the first receptive field to generate the first feature. Figure 1 FM.

[0056] The second variable layer group VG2 may include a second size determination component 2SDU and a second variable convolutional layer 2VCL. The second size determination component 2SDU can be configured to use features included in the first feature... Figure 1 The data in FM compares the image size of the object being processed with the size of the first receptive field used in the first variable convolutional layer 1VCL, and outputs a second compensation value 2VAL as the result of the comparison. For example, the second size determination component 2SDU can output different values ​​of the second compensation value 2VAL when the image size of the object is less than or equal to the size of the first receptive field, and when the image size of the object is greater than the size of the first receptive field. The second variable convolutional layer 2VCL can adjust the kernel size according to the second compensation value 2VAL, and use the adjusted kernel as a filter to process the first feature. Figure 1 FM is used for calculations, and a second feature is generated. Figure 2 FM is the result of the calculation.

[0057] Finally, the Kth variable layer group VGK may include a Kth size determination component KSDU and a Kth variable convolutional layer KVCL. The Kth size determination component KSDU can be configured to compare the image size of the object being processed with the size of the Kth receptive field used in the Kth variable convolutional layer by using data included in the (K-1)th feature map (K-1)FM, and output a Kth compensation value KVAL as the comparison result. For example, if the image size of the object is less than or equal to the size of the K-1 receptive field, and if the image size of the object is greater than the size of the K-1 receptive field, the Kth size determination component KSDU can output different values ​​for the Kth compensation value KVAL. The Kth variable convolutional layer KVCL can adjust the kernel size based on the Kth compensation value KVAL, use the adjusted kernel as a filter to operate on the (K-1)th feature map (K-1)FM, and generate a Kth feature map KFM as the result of the operation.

[0058] As described above, the first variable layer group VG1 to the Kth variable layer group VGK can compare the image size of the previous feature map with the size of the previous receptive field, and determine the size of the next kernel and the next receptive field based on the comparison result. Therefore, the Kth feature map KFM can be generated by using kernel and receptive field sizes optimized for the image size of the object.

[0059] Figure 6A , Figure 6B and Figure 6C This is a diagram that specifically illustrates the method for generating feature maps.

[0060] Reference Figure 6A When the input feature map is configured to a 6×6 size and the kernel is configured to a 3×3 size, the size of the receptive field (RF) can be set to a 3×3 size equal to the kernel size. In the following description, the input feature map is described as the "input feature map," and the feature map generated as a result of operations on the input feature map is described as the "output feature map." The kernel size is a unit of pixels storing the calculated values. The input feature map can be... Figure 5 The convolution operation is input to the layer of the variable network 320. Each pixel of the input feature map can include weight values ​​for the image of the object. Among the pixels of the input feature map, a convolution operation can be performed with the kernel information corresponding to the values ​​of the pixels included in the receptive field RF, and the result can be stored in the first pixel 1PX of the output feature map.

[0061] Reference Figure 6B ,like Figure 6AAs shown, when the result of the operation is stored in the first pixel 1PX of the output feature map, the position of the receptive field RF can be moved one pixel at a time to process the next value. For example, when the result of the operation is stored in the first pixel 1PX, the position of the receptive field RF can be moved one pixel to the right in the input feature map for the next convolution operation. Since the position of the receptive field RF changes in the input feature map, the information included in the changed receptive field RF also changes. Therefore, convolution operations can be performed on the information included in the receptive field RF and the kernel information, and the result can be stored in the second pixel 2PX of the output feature map.

[0062] Reference Figure 6C You can refer to Figure 6A and Figure 6B The described method performs convolution operations on the kernel and the input layer while shifting the position of the receptive field RF. The output feature map, which serves as the last pixel in the output feature map, is completed when the result is stored in the sixteenth pixel (16PX) of the input feature map.

[0063] Figure 7A and Figure 7B This is a diagram illustrating a method for determining variable values ​​for component settings by size according to an embodiment of the present disclosure.

[0064] Reference Figure 7A and Figure 7B The size-determining component SDU can be Figure 5 The diagram shows any one of the first size determination components 1SDU to the Kth size determination component KSDU. Each size determination component SDU compares the size of the object image SI included in the input layer with the size of the receptive field RF and outputs a compensation value VAL as the result of the comparison. In the following description, the size of the object image SI is referred to as the object size SOB, and the size of the receptive field RF is referred to as the receptive field size SRF.

[0065] The object size SOB can be set to the horizontal length W_ob of the object image SI, the vertical length H_ob of the object image SI, or both the horizontal length W_ob and the vertical length H_ob of the object image SI, or the area of ​​the object image SI. When setting the object image SI based on its area, since the size determination component SDU needs to additionally perform calculations to calculate the area of ​​the object image SI, it is preferable to set the object image SI based on either the horizontal length W_ob or the vertical length H_ob.

[0066] The object size SOB may be determined by the values stored in the pixels of the input layer. The receptive field size SRF may be set as the horizontal length W_rf of the receptive field RF, may be set as the vertical length H_rf of the receptive field RF, may be set as both the horizontal length W_rf and the vertical length H_rf of the receptive field RF, or may be set as the area of the receptive field RF. The size determination unit SDU may be configured to compare the same targets of the object size SOB and the receptive field size SRF with each other. For example, when the comparison target for sizes is set to the horizontal length, the size determination unit SDU may compare the horizontal length W_ob of the object image SI with the horizontal length W_rf of the receptive field RF, and output a compensation value VAL as the comparison result.

[0067] In order to output the compensation value VAL from the size determination unit SDU, the size determination unit is trained with a reinforcement learning method, and this learning method is as follows. Initially, the kernel size may be randomly selected for each position of the size determination unit SDU, and the size determination unit SDU may operate the receptive field size SRF using the selected kernel size. For example, the size determination unit SDU may compare the operated receptive field size with the object size SOB, and when the receptive field size is greater than or equal to the object size (SRF≥SOB), a compensation value VAL(-a) may be applied. When the receptive field size operated according to the selected kernel size is smaller than the object size (SRF<SOB), the size determination unit SDU may determine the selected kernel size as an appropriate size and apply a compensation value VAL(+b). Since learning is performed in the direction of increasing the compensation value VAL, the size determination unit SDU is trained to determine the kernel size such that the object size and the receptive field size are similar to each other.

[0068] When the learning of the kernel size is completed, the size determination unit SDU may adjust the compensation value VAL according to the input feature map. For example, the size determination unit SDU may increase the compensation value VAL to enlarge the receptive field size SRF when the object size SOB is large, and may not increase the compensation value VAL when the object size SOB is not large.

[0069] Figure 8A and Figure 8B is a diagram illustrating a method of changing the receptive field when the receptive field is at a minimum value.

[0070] Referring to Figure 8A , it is assumed that the receptive field used for comparing the object size is defined as a first receptive field 1RF, and its size is determined according to Figure 7A or Figure 7BThe size determination component SDU's comparison result adjusts the receptive field to the second receptive field 2RF. When the size of the first receptive field 1RF has a minimum value (here, the size of the first receptive field 1RF may no longer decrease), the object size SOB can be compared with the first receptive field size 1SRF, which is the overall size of the first receptive field 1RF. For example, when the object size SOB is equal to or less than the first receptive field size 1SRF of the first receptive field 1RF (SOB≤1SRF), the core size can be kept the same as the size of the first receptive field 1RF, and therefore the size of the second receptive field 2RF can be kept the same as the size of the first receptive field 1RF. That is, when the size of the first receptive field 1RF is at its minimum value, even if the object size SOB is less than the first receptive field size 1SRF, since the size of the first receptive field 1RF may no longer decrease, a core and a second receptive field 2RF with the same size as the first receptive field 1RF can still be set.

[0071] Reference Figure 8B When the object size SOB is greater than the first acceptor domain size 1SRF of the first acceptor domain 1RF (SOB>1SRF), the size of the core can be set to be greater than the size of the first acceptor domain 1RF, and therefore the size of the second acceptor domain 2RF can be set to be greater than the size of the first acceptor domain 1RF. In other words, when the size of the first acceptor domain 1RF is at its minimum, since the size of the acceptor domain can be adjusted to increase it, a core and a second acceptor domain 2RF with a size greater than the size of the first acceptor domain 1RF can be set.

[0072] As per reference Figure 8A or Figure 8B When the dimensions of the second receptive field 2RF and the kernel are determined as described in the embodiments, a convolution operation can be performed on the information of the input layer included in the determined second receptive field 2RF and the kernel, thereby generating a feature map.

[0073] Figure 9A and Figure 9B This is a diagram illustrating how to change the acceptor domain when it is at its maximum value.

[0074] Reference Figure 9A Assume that the receptive field used for comparing object sizes is defined as a first receptive field 1RF, and its size is based on... Figure 7A or Figure 7BThe receptive domain adjusted by the comparison result of the size determination component SDU is the second receptive domain 2RF. When the size of the first receptive domain 1RF has a maximum value (here, the size of the first receptive domain 1RF may no longer increase), the object size SOB can be compared with the first receptive domain size 1SRF, which is the overall size of the first receptive domain 1RF. For example, when the object size SOB is equal to or greater than the first receptive domain size 1SRF of the first receptive domain 1RF (SOB≥1SRF), the size of the core can be kept the same as the size of the first receptive domain 1RF, and therefore the size of the second receptive domain 2RF can be kept the same as the size of the first receptive domain 1RF. That is, when the size of the first receptive domain 1RF is at its maximum value, even if the object size SOB is equal to or greater than the first receptive domain size 1SRF, since the size of the first receptive domain 1RF cannot be increased, a second receptive domain 2RF with the same size as the first receptive domain 1RF can still be set.

[0075] Reference Figure 9B When the object size SOB is smaller than the first acceptor domain size 1SRF of the first acceptor domain 1RF (SOB < 1SRF), the size of the core can be set to be smaller than the size of the first acceptor domain 1RF, and therefore the size of the second acceptor domain 2RF can be set to be smaller than the size of the first acceptor domain 1RF. In other words, when the size of the first acceptor domain 1RF is at its maximum value, since the size of the acceptor domain can be adjusted to decrease, the size of the second acceptor domain 2RF can be set to be smaller than the size of the first acceptor domain 1RF.

[0076] As per reference Figure 9A or Figure 9B When the dimensions of the second receptive field 2RF and the kernel are determined as described in the embodiments, a convolution operation can be performed on the information of the input layer included in the determined second receptive field 2RF and the kernel, thereby generating a feature map.

[0077] Figure 10A , Figure 10B and Figure 10C This is a diagram illustrating how to change the receptive domain when it is in the middle value.

[0078] Reference Figure 10A Assume that the receptive field used for comparing object sizes is defined as a first receptive field 1RF, and its size is based on... Figure 7A or Figure 7BThe acceptance domain adjusted according to the comparison result of the size determination component SDU is the second acceptance domain 2RF. When the size of the first acceptance domain 1RF has an intermediate value that can be reduced or increased, the object size SOB can be compared with each of the first acceptance domain size 1SRF corresponding to the overall size of the first acceptance domain 1RF and the second acceptance domain size 2SRF which is smaller than the overall size. For example, when the first acceptance domain size 1SRF is set to 100% of the size of the first acceptance domain 1RF, the second acceptance domain size 2SRF can be set to 80% of the size of the first acceptance domain 1RF. The size ratio can be set to various values.

[0079] When the object size SOB is equal to or smaller than the first acceptance domain size 1SRF and larger than the second acceptance domain size 2SRF (2SRF<SOB≤1SRF), the size of the kernel can be maintained at the same size as that of the first acceptance domain 1RF, and thus the size of the second acceptance domain 2RF can be maintained the same as the size of the first acceptance domain 1RF. That is, even if the object size SOB is smaller than the first acceptance domain size 1SRF, when the size difference is not too large, since the size of the first acceptance domain 1RF does not need to be reduced, the second acceptance domain 2RF having the same size as that of the first acceptance domain 1RF can still be set.

[0080] refer to Figure 10B , when the object size SOB is equal to or smaller than the second acceptance domain size 2SRF (SOB≤2SRF), the size of the kernel can be set to be smaller than that of the first acceptance domain 1RF, and thus the size of the second acceptance domain 2RF can be set to be smaller than the size of the first acceptance domain 1RF. That is, when the size of the first acceptance domain 1RF is at an intermediate value and the object size SOB is too small compared with the first acceptance domain size 1SRF, the second acceptance domain 2RF having a size smaller than the size of the first acceptance domain 1RF can be set.

[0081] refer to Figure 10C , when the object size SOB is larger than the first acceptance domain size 1SRF (SOB>1SRF), the size of the kernel can be set to be larger than the size of the first acceptance domain 1RF, and thus the size of the second acceptance domain 2RF can be set to be larger than the size of the first acceptance domain 1RF. That is, when the size of the first acceptance domain 1RF is at an intermediate value, since the size of the acceptance domain can be adjusted to increase, the kernel and the second acceptance domain 2RF having a size larger than the size of the first acceptance domain 1RF can be set.

[0082] When as described with reference to Figure 10A , Figure 10B or Figure 10CWhen the dimensions of the second receptive field 2RF and the kernel are determined as described in the embodiments, a convolution operation can be performed on the information of the input layer included in the determined second receptive field 2RF and the kernel, thereby generating a feature map.

[0083] Figure 11 This is a diagram illustrating a method of operating an image detector according to an embodiment of the present disclosure.

[0084] Reference Figure 11 Image detector 330 can be configured to classify the final generated Kth feature map KFM for each color pair to generate channels, and to generate feature data DATA_FC including the generated channels. That is, in the embodiment, since Figure 5 The variable network 320 adjusts the kernel and receptive field sizes according to the image size of the object, thus preventing excessive enlargement of the receptive field size when the image size is small and excessive shrinkage of the receptive field size when the image size is large. Therefore, the computation time and complexity of the operation used to convert the image of the object into feature data DATA_FC can be improved.

Claims

1. An electronic device, comprising: An image sensor that senses light originating from an object and generates an image layer; as well as Image processor, which: An initial feature map is generated by performing convolution operations on the image layer and a fixed-size kernel; An initial feature map is generated by performing convolution operations on image layers using a fixed-size kernel; Multiple feature maps are generated using multiple variable layer groups, each of which is used for: Compare the size of the object image included in the input feature map with the size of the receptive field; A compensation value is generated based on the comparison results; The size of the variable-size kernel and the size of the receptive field are adjusted based on the compensation value; and An output feature map is generated by performing convolution operations on the input feature map using an adjusted variable-size kernel and an adjusted receptive field. The initial feature map is input to the first variable layer group, and the output feature map of each variable layer group except the last is input to the next variable layer group. The output feature map of the last variable layer group is the final feature map. Image data including channels is generated solely based on the final feature map, which is obtained by classifying the final feature map according to color.

2. The electronic device according to claim 1, wherein, The image sensor includes: A first converter receives a light source from the object and outputs an analog image signal; and A second converter converts the analog image signal into the image layer as a digital signal.

3. The electronic device according to claim 1, wherein, The image processor includes: A fixed-size network generates an initial feature map by performing the convolution operation on the image layer using the fixed-size kernel; A variable network that generates multiple feature maps by performing the convolution operation using the variable-size kernel; An image detector that detects channels for each color classification from the final feature map among the plurality of feature maps, and generates feature data including the channels; and The data output circuit outputs the feature data as is, or outputs the image data by processing, compressing, or encrypting the feature data.

4. The electronic device according to claim 3, wherein, The fixed network generates the initial feature map using the deep convolutional network technique in convolutional neural network technology.

5. The electronic device according to claim 3, wherein, The fixed network: The fixed-size kernel is used as a filter to perform convolution operations; Perform pooling on the result of the convolution operation; and The initial feature map is generated as the result of the pooling.

6. The electronic device according to claim 3, wherein, The variable network includes multiple variable layer groups, which generate the multiple feature maps by using the variable-size kernel as a filter.

7. The electronic device according to claim 1, wherein, The size of the object image and the size of the receiving field are compared based on the corresponding horizontal length, corresponding vertical length, or corresponding area.

8. The electronic device of claim 7, wherein each variable layer group includes a size determination component trained by reinforcement learning to generate a compensation value based on a comparison between the size of the image of the object and the size of the receptive field.

9. The electronic device according to claim 8, wherein, The size-determining component: When the image size of the object included in the input feature map is smaller than the size of the receptive field, the compensation value used to reduce the size of the variable-size kernel is output; as well as When the image size of the object included in the input feature map is larger than the size of the receptive field, the compensation value used to increase the size of the variable-size kernel is output.

10. The electronic device according to claim 8, wherein, When the size of the receiving field is at its minimum, the size determination component: When the image size of the object included in the input feature map is equal to or smaller than the size of the receptive field, the compensation value used to maintain the size of the variable-size kernel is output; as well as When the image size of the object included in the input feature map is larger than the size of the receptive field, the compensation value used to increase the size of the variable-size kernel is output.

11. The electronic device according to claim 8, wherein, When the size of the receiving field is at its maximum value, the size determination component: When the image size of the object included in the input feature map is equal to or greater than the size of the receptive field, the compensation value used to maintain the size of the variable-size kernel is output; as well as When the image size of the object included in the input feature map is smaller than the size of the receptive field, the compensation value used to reduce the size of the variable-size kernel is output.

12. The electronic device according to claim 8, wherein, When the size of the receiving domain is between the minimum and maximum values, the size determination component: When the image size of the object included in the input feature map is included between a first size of the receptive domain and a second size smaller than the first size, the compensation value for maintaining the size of the variable-size kernel is output; When the image size of the object included in the input feature map is equal to or smaller than the second size, the compensation value for reducing the size of the variable-size kernel is output; as well as When the image size of the object included in the input feature map is larger than the first size, the compensation value for increasing the size of the variable-size kernel is output.

13. A method of operating an electronic device, the method comprising: Convert the image of the object into an image layer; Initial feature maps are generated from image layers by performing convolution operations using fixed-size kernels. Multiple feature maps are generated using multiple variable convolutional levels, each of which includes: Compare the size of the object image included in the input feature map with the size of the receptive field; A compensation value is generated based on the comparison results; The size of the variable-size kernel and the size of the receptive field are adjusted based on the compensation value; and An output feature map is generated by performing convolution operations on the input feature map using an adjusted variable-size kernel and an adjusted receptive field. The initial feature map is used as the input feature map in the first variable convolutional level, and the output feature maps generated in each variable convolutional level except the last one are used as the input feature maps in the next variable convolutional level. The output feature map generated in the last variable convolutional level is the final feature map. Channels and feature data including the channels are generated solely based on the final feature map, which is obtained by classifying the final feature map according to color.

14. The method of claim 13, wherein, Converting the image of the object into the image layer includes: Convert the image of the object into an analog image signal; and The analog image signal is converted into the image layer as a digital signal.

15. The method according to claim 13, wherein, Generating the image layer as the initial feature map and generating the multiple feature maps are performed using a convolutional neural network method.

16. The method according to claim 13, wherein, Generating the image layer as the initial feature map includes: Convolutional layers are generated by performing the convolution operation using the fixed-size kernel; and The initial feature map is generated by pooling the convolutional layer.

17. The method according to claim 16, wherein, In generating the convolutional layer, a receptive field with a fixed size is used.

18. The method according to claim 13, wherein, The size of the object image and the size of the receiving field are compared based on the corresponding horizontal length, corresponding vertical length, or corresponding area.

19. The method according to claim 18, wherein, The size of the receiving field is adjusted according to the size of the variable-size core.

20. The method according to claim 19, wherein, In generating the multiple feature maps, the feature map generated in the previous step will be used as the input feature map for the next step.

Images