Electronic device including image processing unit for removing object corresponding to rainwater from image and driving method thereof
The transformer-based deep learning operation in the electronic device efficiently removes rainwater from images by using a multi-scale hierarchical structure, addressing irregular shapes and reducing GPU memory, thus enhancing image processing efficiency.
Patent Information
- Application Number
- KR1020240173520
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2044-11-28
AI Technical Summary
Conventional methods for removing rainwater from images struggle with effectively removing irregularly shaped rainwater while maintaining important image details, and they require significant GPU memory during high-resolution processing.
An electronic device with a transformer-based deep learning operation that includes a rainwater removal unit with a multi-scale hierarchical structure and a distortion correction unit, utilizing transformer modules and convolutional layers to reduce image resolution, process rainwater removal, and correct image quality.
Effectively removes rainwater of various shapes while reducing GPU memory requirements, maintaining image quality, and improving processing efficiency.
Smart Images

Figure 112024131823828-PAT00003_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to an electronic device comprising an image processing unit for removing an object corresponding to rainwater from an image, and a method for operating the same. Background Technology
[0002] Rainwater can obscure important objects or scenes in images, potentially distorting or damaging visual information. In particular, in various vision-based application systems (e.g., electronic devices) such as autonomous driving, surveillance systems (CCTV), and drone photography, images containing rainwater can hinder analysis—including object recognition, tracking, and classification—and degrade performance. Therefore, technology for removing rainwater from images plays a crucial role in improving the performance of various vision-based applications by eliminating unnecessary information to generate clean images.
[0003] Various methods for removing rainwater from images have been researched and developed, and filtering-based and geometric approaches have been introduced. Filtering-based approaches use low-intensity or high-intensity filters (e.g., Gaussian filters, median filters, etc.) to remove high-frequency components containing rainwater, while geometric approaches model the shape of the rainwater and analyze the movement, direction, and velocity of raindrops to remove it.
[0004] However, conventional methods for removing rainwater from images had limitations in the performance of removing objects corresponding to the rainwater. For example, filtering-based approaches suffered from the problem of removing important details along with the rainwater during the filtering process, while geometric approaches had limitations in handling the complex and irregular movements of raindrops. The problem to be solved
[0005] Various embodiments of the present invention aim to solve the above problems and provide an electronic device including an image processing unit that performs a transformer-based deep learning operation capable of effectively removing irregularly shaped rainwater while reducing GPU memory requirements during high-resolution image processing, and a method for operating the same.
[0006] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem
[0007] An electronic device according to one embodiment of the present disclosure includes a control unit, a camera module, a memory, and an image processing unit, wherein the image processing unit includes a rainwater removal unit that receives a first image as an input image and generates a second image by removing at least one object corresponding to rainwater included in the received first image, and a distortion correction unit that receives the second image, generates a third image by correcting the color and quality of the input second image, and outputs the generated third image as a final image, wherein the rainwater removal unit may include an image resolution reduction module that reduces the resolution of the first image and increases the number of channels of the first image, a rainwater removal core module that includes a plurality of transformer modules in a multi-scale hierarchical structure and performs at least two size reduction processes and at least two size expansion processes, and an image expansion module that converts the resolution of the image reduced by the image resolution reduction module to the resolution of the first image and generates the second image by synthesizing the converted image and the first image.
[0008] A method for removing rainwater included in an image using an electronic device according to one embodiment of the present disclosure comprises: a rainwater removal unit receiving a first image as an input image and generating a second image by removing at least one object corresponding to rainwater included in the received first image; and a distortion correction unit receiving the second image and generating a third image by correcting the color and quality of the input second image, and outputting the generated third image as a final image. The rainwater removal unit may include an image resolution reduction module that reduces the resolution of the first image and increases the number of channels of the first image; a rainwater removal core module that includes a plurality of transformer modules in a multi-scale hierarchical structure and performs at least two size reduction processes and at least two size expansion processes; and an image expansion module that converts the resolution of the image reduced by the image resolution reduction module to the resolution of the first image and synthesizes the converted image and the first image to generate the second image. Effects of the invention
[0009] According to embodiments of the present disclosure, rainwater of various shapes (e.g., structured or irregular shapes) can be effectively removed while reducing GPU memory requirements during high-resolution image processing.
[0010] In addition, various effects that can be identified directly or indirectly through this document may be provided. Brief explanation of the drawing
[0011] Other aspects, features, and advantages according to specific embodiments of the present disclosure will become more apparent from the accompanying drawings and description. FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment. FIG. 2 is a conceptual diagram schematically illustrating a method in which an image processing unit according to one embodiment removes an object corresponding to rainwater from an image. FIG. 3 is a block diagram showing the detailed configuration of a rainwater removal unit according to one embodiment. FIG. 4 is a block diagram showing the detailed configuration of a transformer module according to one embodiment. FIG. 5 is a block diagram showing the detailed configuration of a multi-head attention module according to one embodiment. FIG. 6 is a block diagram showing the detailed structure of a feed-forward module according to one embodiment. FIG. 7 is a block diagram showing the detailed configuration of a distortion correction unit according to one embodiment. FIG. 8 is a flowchart illustrating a method of driving an electronic device according to one embodiment. Specific details for implementing the invention
[0012] The aforementioned objectives, features, and advantages are described in detail below with reference to the attached drawings, thereby enabling those skilled in the art to easily implement the technical concept of the present invention. In describing the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such descriptions would unnecessarily obscure the essence of the invention. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings.
[0013] FIG. 1 is a block diagram showing the components of an electronic device (10) according to one embodiment of the present invention.
[0014] Referring to FIG. 1, an electronic device (10) according to one embodiment may include a control unit (101), a memory (103), a camera module (105), a communication module (107), and an image processing unit (109). The image processing unit (109) may include a rainwater removal unit (120) and a distortion correction unit (140).
[0015] Unlike the illustrated example, the electronic device (10) according to one embodiment may not include at least some of the components illustrated in FIG. 1. Unlike the illustrated example, the electronic device (10) according to one embodiment may further include components not illustrated in FIG. 1 but widely and generally known in the art.
[0016] The electronic device (10) according to the various embodiments disclosed in this disclosure may be of various forms. The electronic device (10) may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device (10) may be a wearable device. A wearable device may include a head-mounted display (HMD) that is wearable on a user's head. A wearable device may be referred to as a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. The electronic device (10) according to the embodiments of this disclosure is not limited to the aforementioned devices.
[0017] According to one embodiment, the control unit (101) may include at least one processor. The processor may, for example, execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of the electronic device (10) connected to the processor and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor may store commands or data received from other components (e.g., a sensor module or a communication module (107)) in volatile memory, process the commands or data stored in volatile memory, and store the resulting data in non-volatile memory. According to one embodiment, the processor may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor).
[0018] According to one embodiment, the memory (103) may store various data used by at least one component of the electronic device (10) (e.g., a processor or an image processing unit (109)). The data may include, for example, software (e.g., a program) and input data or output data for related instructions. The memory (103) may include volatile memory or non-volatile memory.
[0019] According to one embodiment, the camera module (105) can capture still images and video. According to one embodiment, the camera module (105) may include one or more lenses, image sensors, image signal processors, or flashes.
[0020] According to one embodiment, the communication module (107) may support the establishment of a wired communication channel or a wireless communication channel between an electronic device (10) and an external electronic device (e.g., a server), and the performance of communication through the established communication channel. According to one embodiment, the communication module (107) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (e.g., a LAN (local area network) communication module, or a power line communication module).
[0021] According to one embodiment, the image processing unit (109) may be configured to remove objects corresponding to rainwater from the image. The image processing unit (109) may receive an image from the camera module (105) (or memory (103)) and output a final image with rainwater removed by processing and correcting the received image. To this end, the image processing unit (109) may include a rainwater removal unit (120) and a distortion correction unit (140), and their operations will be described in detail later with reference to FIGS. 2 to 7. According to one embodiment, at least some functions of the image processing unit (109) may be performed by the control unit (101). For example, at least some operations performed by the image processing unit (109) may be performed directly by the control unit (101).
[0022] According to one embodiment, the image processing unit (109) may include an artificial intelligence model for image processing. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (10) itself where the artificial intelligence model is performed, or through a separate server (e.g., a server). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. Artificial intelligence models may include software structures, either additionally or as a substitute, in addition to hardware structures.
[0023] The term “image processing unit (109)” as used in various embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as module, logic, logic block, component, or circuit, for example. A module may be a component formed integrally or a minimum unit of said component or a part thereof that performs one or more functions. For example, a module may be implemented in the form of an application-specific integrated circuit (ASIC). The term “module” as used in various embodiments of the present disclosure may be used interchangeably with terms such as “block”.
[0024] FIG. 2 is a conceptual diagram schematically illustrating a method in which an image processing unit (109) according to one embodiment removes an object corresponding to rainwater from an image.
[0025] According to one embodiment, the image processing unit (109) may be composed of two main modules (120, 140), and may include, for example, a rainwater removal unit (120) and a distortion correction unit (140).
[0026] According to one embodiment, the rainwater removal unit (120) receives a first image (110) as an input image and can generate a second image (130) by removing at least one object corresponding to rainwater included in the received first image (110). The rainwater removal unit (120) outputs the generated second image (130), and the second image (130) output by the rainwater removal unit (120) can be input to a distortion correction unit (140). According to one embodiment, the rainwater removal unit (120) can perform the role of removing rainwater (e.g., an object corresponding to rainwater) included in the first image (110) as a deraining module.
[0027] According to one embodiment, the distortion correction unit (140) receives a second image (130) and can generate a third image (150) by correcting the color and quality of the input second image (130). The distortion correction unit (140) can output the generated third image (150) as a final image. According to one embodiment, the distortion correction unit (140) can function as a color distortion correction module to correct color distortion and image quality degradation caused by rain, fog, etc.
[0028] FIG. 3 is a block diagram showing the detailed configuration of a rainwater removal unit (120) according to one embodiment. For example, the rainwater removal unit (120) shown in FIG. 3 may be substantially the same as the rainwater removal unit (120) shown in FIG. 2.
[0029] Referring to FIG. 3, the rainwater removal unit (120) receives a first image (210) as an input image and can generate a second image (260) by removing at least one object corresponding to rainwater included in the received first image (210). The rainwater removal unit (120) can output the generated second image (260). The first image (210) shown in FIG. 3 may be substantially identical to the first image (110) shown in FIG. 2. The second image (260) shown in FIG. 3 may be substantially identical to the second image (130) shown in FIG. 2.
[0030] According to one embodiment, the rainwater removal unit (120) may include an image resolution reduction module (220), a rainwater removal core module (230), and an image resolution expansion module (240).
[0031] According to one embodiment, an image resolution reduction module (220) may be used to reduce computational load and increase processing speed. The image resolution reduction module (220) may be composed of a single convolution layer. For example, the image resolution reduction module (220) may be represented as C4S2-d and may be configured to use a 4x4 filter and a stride of 2 to reduce the resolution (size) of an input image by half and increase the number of channels from 3 (R, G, B) to d.
[0032] According to one embodiment, the rainwater removal core module (230) may include a plurality of transformer modules (231) in a multi-scale hierarchical structure. To implement the hierarchical structure, the rainwater removal core module (230) may perform at least two size reduction (232, 233) and at least two size expansion (234, 235) processes, and may use a convolutional layer and a transposed convolution layer.
[0033] The first size reduction (232) performed by the rainwater removal core module (230) may use a convolutional layer (C4S2-2d) with a stride of 2 and a 4x4 filter, and the number of output channels may be increased to 2d. Here, d represents the number of channels of information transmitted to the rainwater removal core module (the number of output channels of the image resolution reduction module), and is intended to reduce the size by half and double the number of channels.
[0034] The second size reduction (233) performed by the rainwater removal core module (230) can utilize a convolutional layer (C4S2-4d) with the same stride and filter size as the first size reduction (232), and the number of output channels can be increased to 4d. In the second size reduction (233) process, the size can be reduced by half again and the number of channels can be doubled.
[0035] In the illustrated example, the rainwater removal core module (230) is described as performing two size reduction processes (232, 233), but the rainwater removal core module (230) may perform two or more size reduction processes. For example, the rainwater removal core module (230) may perform three or four size reduction processes. The number of times the rainwater removal core module (230) performs size reduction may be set to two or more values based on results learned by artificial intelligence.
[0036] The first size expansion (234) performed by the rainwater removal core module (230) can use an inverse convolutional layer (DC4S2-2d) using a 2x2 filter with a stride of 2, and the size can be expanded by a factor of two and the number of channels reduced by a factor of half.
[0037] The second size expansion (235) performed by the rainwater removal core module (230) can use an inverse convolutional layer (DC4S2-d) using a 2x2 filter with a stride of 2, and the size can be expanded by a factor of two and the number of channels reduced by a factor of half.
[0038] In the illustrated example, the rainwater removal core module (230) is described as performing two size expansion (234, 235) processes, but the rainwater removal core module (230) may perform two or more size expansion processes. For example, the rainwater removal core module (230) may perform three size expansion processes or four size expansion processes. The number of times the rainwater removal core module (230) performs size expansion may be set to two or more values based on results learned by artificial intelligence.
[0039] In FIG. 3, C represents a Concat operation, and after the Concat operation, a convolution operation (C1S1-2d) can be performed to adjust the number of channels doubled by the Concat operation using a 1x1 filter with a stride of 1.
[0040] According to one embodiment, the image expansion module (240) may be configured to expand the resolution (size) of the image reduced by the image resolution reduction module (220) to the size of the original image (e.g., the first image). The image expansion module (240) may include a single convolutional layer and a pixel shuffle layer represented by C3S1-12. C3S1-12 uses a 3x3 filter with a stride of 1 and 12 output channels, which may be the result (3x2x2) of considering the number of output channels (R, G, B) of the pixel shuffle layer and the horizontal and vertical expansion ratios (2x each). The pixel shuffle layer may expand the resolution of the image transmitted from the rainwater removal core module (230) to the same resolution as the first image (110) by using a super-resolution technique that converts the low-resolution image into a high-resolution image. According to one embodiment, the image expansion module (240) can convert the resolution of the image transmitted from the rainwater removal core module (230) to the same resolution (size) and number of channels as the resolution of the first image (110), and can generate a second image (130) by synthesizing (250) the image transmitted from the rainwater removal core module (230) and the first image (110).
[0041] FIG. 4 is a block diagram showing the detailed configuration of a transformer module according to one embodiment. For example, the transformer module shown in FIG. 4 may be substantially the same as the transformer module shown in FIG. 3.
[0042] Referring to FIG. 4, the transformer module (231, 310) may include a multi-head attention module (320) and a feed forward module (330).
[0043] According to one embodiment, a multi-head attention module (320) can be designed to effectively process global information of the entire image, and can be configured to process multiple head attention modules (320) in parallel (341) to learn and process various patterns.
[0044] According to one embodiment, the feed-forward module (330) can learn and process more complex patterns by non-linearly transforming the information that has passed through the multi-head attention module (320).
[0045] According to one embodiment, the transformer module (231, 310) may use a layer normalization module (340) and a residual connection (350) to support stable learning of the multi-head attention module (320) and the feed-forward module (330).
[0046] FIG. 5 is a block diagram showing the detailed configuration of a multi-head attention module according to one embodiment. For example, the multi-head attention module illustrated in FIG. 5 may be substantially the same as the multi-head attention module illustrated in FIG. 4.
[0047] According to one embodiment, the input of the multi-head attention module (320) may be transmitted in the form of an embedding vector. As a first step, the multi-head attention module (320) inputs an input embedding vector I L From (410), query (Q), key (K), and value (V) values can be calculated. To do this, the multi-head attention module (320) input embedding vector I L(410) weight matrix W q , W k , W v The values of Q (411), K (412), and V (413) can be calculated by multiplying them respectively. In FIG. 5, the weight matrix W q , W k , W v The multiplication operation is represented by a Linear block (414), and each weight matrix can be configured to be optimized during the learning process.
[0048] According to one embodiment, the multi-head attention module (320) can perform an attention (420) process as a second step. The attention (420) process can be expressed mathematically as Equation 1. Referring to Equation 1, the multi-head attention module (320) can calculate the attention weight through a softmax function after calculating the inner product of Q and K. To prevent the inner product value from becoming excessively large during the process of calculating the weight, the multi-head attention module (320) uses a square root of the vector dimension The value can be adjusted. The attention weight represents the relative importance between input elements and can be a factor in determining how much attention should be paid to each element. The multi-head attention module (320) can finally produce a final result by multiplying the attention weight by the value (V).
[0049]
[0050] In FIG. 5, the attention (attention, 420) process performed by the multi-head attention module (320) is represented in module units (421), the inner product of Q and K is represented as MatMul (422), the operation of dividing by the square root of the vector dimension is represented as Normalization (423), the softmax function is represented as Softmax (424), and the operation of multiplying the attention weight by the value is represented as MatMul (425).
[0051] According to one embodiment, the multi-head attention module (320) may require a process of combining each result into one as it processes a plurality of attention modules (420) in parallel. The multi-head attention module (320) combines the result values of the plurality of attention (attention, 420) processes processed in parallel into one through a Concat (430) operation, and then combines the input embedding vector I through a Linear (440) operation. L The result information can be adjusted with the same size and number of channels as (410).
[0052] FIG. 6 is a block diagram showing the detailed structure of a feed-forward module according to one embodiment. For example, the feed-forward module shown in FIG. 6 may be substantially the same as the feed-forward module (330) shown in FIG. 4.
[0053] According to one embodiment, the feed-forward module (330) can perform the role of learning and processing more complex patterns by non-linearly converting information (i.e., images) that has passed through the multi-head attention module (320).
[0054] According to one embodiment, the feed-forward module (330) may include three convolutional layers (510, 530, 550) and two GELU activation functions (520, 540).
[0055] According to one embodiment, in the feed-forward module (330), the first convolutional layer C1S1-k (510) can increase the number of channels from d to k while maintaining the resolution (size) of the image using a 1x1 filter with a stride of 1. Here, d is the number of output channels of the multi-head attention module, and k can be any number specified by the user that is greater than d.
[0056] According to one embodiment, the feed-forward module (330) can increase the number of channels while maintaining the resolution (size) of the image, and then apply the first GELU activation function (520) to add non-linearity to the information (i.e., image) that has passed through the multi-head attention module (320). According to one embodiment, the feed-forward module (330) can improve model performance by providing more efficient non-linear characteristics compared to general activation functions such as RELU or Leaky RELU.
[0057] According to one embodiment, in the feedforward module (330), the second convolutional layer, DC3S1-k (530), is a depthwise convolutional layer using a 3x3 filter with a stride of 1, and can reduce memory (103) usage and computational load while maintaining performance by processing convolutions independently for each channel.
[0058] According to one embodiment, the feed-forward module (330) can further enhance non-linearity by applying a second GELU activation function (540) after the processing step by DC3S1-k (530).
[0059] According to one embodiment, the feedforward module (330) can be adjusted to the same number of channels as the input of the feedforward module (330) using a 1x1 filter of the convolutional layer C1S1-d (550) after applying the GELU activation function (540).
[0060] According to various embodiments of the present disclosure, the rainwater removal unit (120) can remove rainwater in a low-resolution image space with reduced image resolution (size) to reduce processing time. However, image quality degradation may occur during this process. Additionally, the image from which rainwater has been removed may exhibit color distortion due to the characteristics of the rainy environment when compared to an image taken in clear weather. To solve these problems, the present invention may additionally include a distortion correction unit (140) for correcting image quality degradation and color distortion. Hereinafter, the distortion correction unit (140) will be described in detail with reference to FIG. 7.
[0061] FIG. 7 is a block diagram showing the detailed configuration of a distortion correction unit (140) according to one embodiment. For example, the distortion correction unit (140) shown in FIG. 7 may be substantially the same as the distortion correction unit (140) shown in FIG. 2.
[0062] According to one embodiment, the distortion correction unit (140) may receive a second image (410) and generate a third image (440) by correcting the color and quality of the input second image (410). The distortion correction unit (140) may output the generated third image (440) as a final image. The second image (410) illustrated in FIG. 7 may be substantially identical to the second image (130) illustrated in FIG. 2. The third image (440) illustrated in FIG. 7 may be substantially identical to the third image (150) illustrated in FIG. 2.
[0063] According to one embodiment, the distortion correction unit (140) may be at least partially similar to the rainwater removal unit (120) described with reference to FIG. 3. According to one embodiment, the distortion correction unit (140) may include an image resolution reduction module (450), an image correction core module (460), and an image resolution expansion module (470).
[0064] According to one embodiment, the image resolution reduction module (450) can reduce the resolution (size) of the second image (410) (e.g., input image) and increase the number of channels, and then transmit the information to the image correction core module (460).
[0065] According to one embodiment, the image correction core module (460) is configured with a multi-scale hierarchical structure and may include a plurality of transformer modules (461). The result calculated in the image correction core module (460) may be transmitted to the image resolution expansion module (470).
[0066] According to one embodiment, the image expansion module (470) can convert the resolution and number of channels of the transmitted result to be the same as the second image (130). The image expansion module (470) can generate a third image (150) by synthesizing the converted image and the second image (130).
[0067] According to one embodiment, the image correction core module (460) may include substantially the same components as the rainwater removal core module (230) described with reference to FIG. 3. For example, the image correction core module (460) may include a plurality of transformer modules (461) in a multi-scale hierarchical structure. To implement the hierarchical structure, the image correction core module (460) may perform at least two size reduction and at least two size expansion processes and may use a convolutional layer and a transposed convolution layer.
[0068] FIG. 8 is a flowchart illustrating a method of driving an electronic device (10) according to one embodiment.
[0069] The operations illustrated in FIG. 8 can be performed by instructions stored in memory (103) (e.g., memory (103) of FIG. 1). For example, the instructions can cause the electronic device (10) (e.g., electronic device (10) of FIG. 1) to perform the operations illustrated in FIG. 8 when executed by the control unit (101) (e.g., control unit (101) of FIG. 1).
[0070] In operation 810, the image processing unit (109) can receive the first image (110). For example, the image processing unit (109) can receive the first image (110) captured through the camera module (105). For example, the image processing unit (109) can receive the first image (110) stored in the memory (103).
[0071] In operation 820, the rainwater removal unit (120) of the image processing unit (109) receives the first image (110) as an input image and can generate the second image (130) by removing at least one object corresponding to rainwater included in the received first image (110).
[0072] According to one embodiment, as described with reference to FIG. 3, the rainwater removal unit (120) may include an image resolution reduction module (220), a rainwater removal core module (230), and an image resolution expansion module (240).
[0073] According to one embodiment, an image resolution reduction module (220) may be used to reduce computational load and increase processing speed. The image resolution reduction module (220) may be composed of a single convolution layer. For example, the image resolution reduction module (220) may be represented as C4S2-d and may be configured to use a 4x4 filter and a stride of 2 to reduce the resolution (size) of an input image (e.g., a first image) by half and increase the number of channels from 3 (R, G, B) to d.
[0074] According to one embodiment, the rainwater removal core module (230) may include a plurality of transformer modules (231) in a multi-scale hierarchical structure. To implement the hierarchical structure, the rainwater removal core module (230) may perform at least two size reduction (232, 233) and at least two size expansion (234, 235) processes, and may use a convolutional layer and a transposed convolution layer.
[0075] In operation 830, according to one embodiment, the distortion correction unit (140) receives a second image (130) and can generate a third image (150) by correcting the color and quality of the input second image (130). The distortion correction unit (140) can output the generated third image (150) as a final image. According to one embodiment, the distortion correction unit (140) can function as a color distortion correction module to correct color distortion and image quality degradation caused by rain, fog, etc.
[0076] According to one embodiment, as described with reference to FIG. 7, the distortion correction unit (140) may include an image resolution reduction module (450), an image correction core module (460), and an image resolution expansion module (470).
[0077] According to one embodiment, the image resolution reduction module (450) can reduce the resolution (size) of the input image (610) and increase the number of channels, and then transmit the information to the image correction core module (460).
[0078] According to one embodiment, the image correction core module (460) is configured with a multi-scale hierarchical structure and may include multiple transformer modules. The result calculated in the image correction core module (460) may be transmitted to the image resolution expansion module (470).
[0079] According to one embodiment, the image expansion module (470) can convert the resolution and number of channels of the transmitted result to be the same as the second image (130). The image expansion module (470) can generate a third image (150) by synthesizing the converted image and the second image (130).
[0080] An electronic device according to one embodiment of the present disclosure includes a control unit, a camera module, a memory, and an image processing unit, wherein the image processing unit includes a rainwater removal unit that receives a first image as an input image and generates a second image by removing at least one object corresponding to rainwater included in the received first image, and a distortion correction unit that receives the second image, generates a third image by correcting the color and quality of the input second image, and outputs the generated third image as a final image, wherein the rainwater removal unit may include an image resolution reduction module that reduces the resolution of the first image and increases the number of channels of the first image, a rainwater removal core module that includes a plurality of transformer modules in a multi-scale hierarchical structure and performs at least two size reduction processes and at least two size expansion processes, and an image expansion module that converts the resolution of the image reduced by the image resolution reduction module to the resolution of the first image and generates the second image by synthesizing the converted image and the first image.
[0081] The first size reduction performed by the above-mentioned rainwater removal core module is a process of reducing the size by half and doubling the number of channels by using a convolutional layer (C4S2-2d) with a stride of 2 and a 4x4 filter, and the second size reduction performed by the above-mentioned rainwater removal core module is a process of reducing the size by half and doubling the number of channels by using a convolutional layer (C4S2-4d) with the same stride and filter size, and the first size expansion performed by the above-mentioned rainwater removal core module is a process of doubling the size and halving the number of channels by using an inverse convolutional layer (DC4S2-2d) with a stride of 2 and a 2x2 filter, and the second size expansion performed by the above-mentioned rainwater removal core module is a process of doubling the size and halving the number of channels by using an inverse convolutional layer (DC4S2-d) with a stride of 2 and a 2x2 filter.
[0082] The image expansion module described above includes one convolutional layer and a pixel shuffle layer, and the convolutional layer may be configured to use a 3x3 filter with a stride of 1 and have 12 output channels, taking into account the number of output channels of the pixel shuffle layer and the horizontal and vertical expansion ratios.
[0083] The Pixel Shuffle layer can be configured to use a super-resolution technique that converts a low-resolution image into a high-resolution image, thereby expanding the resolution of the image transmitted from the rainwater removal core module (230) to the same resolution as the first image.
[0084] The plurality of transformer modules may include a multi-head attention module configured to process global information of the entire image, and a feed forward module configured to non-linearly transform information that has passed through the multi-head attention module.
[0085] The above multi-head attention module, as a first step, input embedding vector I L Input embedding vector I to calculate query (Q), key (K), and value (V) values from L weight matrix W q , W k , W v Q (411), K (412), and V (413) values are calculated by multiplying them respectively, and as a second step, the inner product of Q and K is calculated, and then attention weights are calculated through a softmax function, and the calculated attention weights are multiplied by the value (V) to produce the final result.
[0086] The above feed-forward module can be configured to increase the number of channels while maintaining the resolution of the image, and then apply a GELU activation function to add non-linearity to the information that has passed through the multi-head attention module.
[0087] A method for removing rainwater included in an image using an electronic device according to one embodiment of the present disclosure comprises: a rainwater removal unit receiving a first image as an input image and generating a second image by removing at least one object corresponding to rainwater included in the received first image; and a distortion correction unit receiving the second image and generating a third image by correcting the color and quality of the input second image, and outputting the generated third image as a final image. The rainwater removal unit may include an image resolution reduction module that reduces the resolution of the first image and increases the number of channels of the first image; a rainwater removal core module that includes a plurality of transformer modules in a multi-scale hierarchical structure and performs at least two size reduction processes and at least two size expansion processes; and an image expansion module that converts the resolution of the image reduced by the image resolution reduction module to the resolution of the first image and synthesizes the converted image and the first image to generate the second image.
[0088] The first size reduction performed by the above-mentioned rainwater removal core module is a process of reducing the size by half and doubling the number of channels by using a convolutional layer (C4S2-2d) with a stride of 2 and a 4x4 filter, and the second size reduction performed by the above-mentioned rainwater removal core module is a process of reducing the size by half and doubling the number of channels by using a convolutional layer (C4S2-4d) with the same stride and filter size, and the first size expansion performed by the above-mentioned rainwater removal core module is a process of doubling the size and halving the number of channels by using an inverse convolutional layer (DC4S2-2d) with a stride of 2 and a 2x2 filter, and the second size expansion performed by the above-mentioned rainwater removal core module is a process of doubling the size and halving the number of channels by using an inverse convolutional layer (DC4S2-d) with a stride of 2 and a 2x2 filter.
[0089] The image expansion module described above includes one convolutional layer and a pixel shuffle layer, and the convolutional layer may be configured to use a 3x3 filter with a stride of 1 and have 12 output channels, taking into account the number of output channels of the pixel shuffle layer and the horizontal and vertical expansion ratios.
[0090] According to various embodiments of the present invention, it is possible to perform transformer-based rainwater removal while maintaining performance and reducing computational complexity and processing time. The computational complexity of the transformer increases in proportion to the square of the product of the image width and height. Therefore, if the image size is reduced by half, the amount of computation can be reduced by approximately 1 / 16. The present invention solves the problem caused by reducing the image size by half by adding a distortion correction unit (140), and by using two modules (e.g., 120 and 140 in FIG. 2), the amount of computation can be reduced by approximately 1 / 8 compared to the existing transformer-based method while maintaining similar performance.
[0091] Therefore, compared to existing transformer-based methods, the present invention can significantly reduce the amount of computation and GPU memory (103) required while maintaining performance. As a result, it is expected to be applicable to various vision-based application systems, such as intelligent CCTV, autonomous driving, and drones, where GPU capacity is limited.
[0092] Although the present invention has been illustrated and described above in relation to a specific preferred embodiment, it is obvious to those skilled in the art that the present invention can be modified and varied in various ways without departing from the technical features or scope of the invention as defined by the following claims. Explanation of the symbols
[0093] 10: Electronic devices 101: Control unit 103: Memory 105: Camera module 107: Communication Module 109: Image processing unit 120: Rainwater removal unit 140: Distortion correction section
Claims
Claim 1 In an electronic device, a control unit; a camera module; a memory; The system includes an image processing unit, wherein the image processing unit comprises: a rainwater removal unit that receives a first image as an input image and generates a second image by removing at least one object corresponding to rainwater included in the received first image; and a distortion correction unit that receives the second image, generates a third image by correcting the color and quality of the input second image, and outputs the generated third image as a final image. The rainwater removal unit comprises: an image resolution reduction module that reduces the resolution of the first image and increases the number of channels of the first image; a rainwater removal core module that includes a plurality of transformer modules in a multi-scale hierarchical structure and performs at least two size reduction processes and at least two size expansion processes; and an image expansion module that converts the resolution of the image reduced by the image resolution reduction module to the resolution of the first image and generates the second image by synthesizing the converted image and the first image. The first size reduction performed by the rainwater removal core module is a convolutional layer (C4S2-) using a 4x4 filter with a stride of 2. 2d It is a process of reducing the size by half and doubling the number of channels by using ), and the second size reduction performed by the above-mentioned rainwater removal core module is a convolutional layer (C4S2-) using the same stride and filter size. 4d By using ), it is a process of reducing the size by half and doubling the number of channels, and the first size expansion performed by the above-mentioned rainwater removal core module is an inverse convolutional layer (DC4S2-) using a 2x2 filter with a stride of 2. 2d By using ), the size is doubled and the number of channels is halved; and the second size expansion performed by the aforementioned rainwater removal core module is an inverse convolutional layer (DC4S2-) using a 2x2 filter with a stride of 2. d An electronic device, which is a process of doubling the size and halving the number of channels by using ). Claim 2 delete Claim 3 An electronic device according to claim 1, wherein the image expansion module comprises one convolutional layer and a pixel shuffle layer, and the convolutional layer uses a 3x3 filter with a stride of 1 and is configured such that the number of output channels becomes 12, taking into account the number of output channels of the pixel shuffle layer and the horizontal and vertical expansion ratios. Claim 4 An electronic device according to claim 3, wherein the Pixel Shuffle layer is configured to expand the resolution of the image transmitted from the rainwater removal core module (230) to the same resolution as the resolution of the first image by using a super-resolution technique that converts a low-resolution image into a high-resolution image. Claim 5 An electronic device according to claim 1, wherein the plurality of transformer modules comprises a multi-head attention module configured to process global information of the entire image, and a feed forward module configured to non-linearly transform information that has passed through the multi-head attention module. Claim 6 In claim 5, the multi-head attention module, as a first step, has an input embedding vector I L Input embedding vector I to calculate query (Q), key (K), and value (V) values from L weight matrix W q , W k , W v An electronic device that calculates the values of Q (411), K (412), and V (413) by multiplying each of them, and as a second step, calculates the inner product of Q and K, calculates the attention weight through a softmax function, and calculates the final result by multiplying the calculated attention weight by the value (V). Claim 7 An electronic device according to claim 5, wherein the feed-forward module is configured to increase the number of channels while maintaining the resolution of the image, and then apply a GELU activation function to add non-linearity to the information passed through the multi-head attention module. Claim 8 A method for an electronic device to remove rainwater contained in an image comprises: a rainwater removal unit receiving a first image as an input image and generating a second image by removing at least one object corresponding to rainwater contained in the received first image; and a distortion correction unit receiving the second image and generating a third image by correcting the color and quality of the input second image, and outputting the generated third image as a final image. The rainwater removal unit comprises an image resolution reduction module that reduces the resolution of the first image and increases the number of channels of the first image; a rainwater removal core module that includes a plurality of transformer modules in a multi-scale hierarchical structure and performs at least two size reduction processes and at least two size expansion processes; and an image expansion module that converts the resolution of the image reduced by the image resolution reduction module to the resolution of the first image and synthesizes the converted image and the first image to generate the second image. The first size reduction performed by the rainwater removal core module is a convolutional layer (C4S2-) using a 4x4 filter with a stride of 2. 2d It is a process of reducing the size by half and doubling the number of channels by using ), and the second size reduction performed by the above-mentioned rainwater removal core module is a convolutional layer (C4S2-) using the same stride and filter size. 4d By using ), it is a process of reducing the size by half and doubling the number of channels, and the first size expansion performed by the above-mentioned rainwater removal core module is an inverse convolutional layer (DC4S2-) using a 2x2 filter with a stride of 2. 2d By using ), the size is doubled and the number of channels is halved; and the second size expansion performed by the aforementioned rainwater removal core module is an inverse convolutional layer (DC4S2-) using a 2x2 filter with a stride of 2. d A method that doubles the size and reduces the number of channels by half by using ). Claim 9 delete Claim 10 In claim 8, the image expansion module comprises one convolutional layer and a pixel shuffle layer, wherein the convolutional layer uses a 3x3 filter with a stride of 1 and is set so that the number of output channels becomes 12, taking into account the number of output channels of the pixel shuffle layer and the horizontal and vertical expansion ratios.
Citation Information
Patent Citations
Apparatus and method for improving images in poor weather
KR102514531B1
