Image processing method and electronic equipment

Abstract

The invention provides an image processing method and electronic equipment. The image processing method includes: selecting a first sensor from among a plurality of sensors; processing the output data of the first sensor using a first sensor-specific signal processing having a first individual setting dedicated to the first sensor and a sensor-independent signal processing having a common setting of the plurality of sensors; and executing the first task based on the processed output data.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2023-0141363 filed on October 20, 2023, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following description relates to an image processing method and an electronic device. Background Art

[0003] In digital imaging, the sensor obtains light from a spectrum passing through a color filter array (CFA). For example, a common CFA is a Bayer pattern with two green (G) channels, one red (R) channel, and one blue (B) channel arranged in a repeating 2×2 pattern. The Bayer pattern is designed based on human temporal perception capabilities. In addition, the output of the sensor (e.g., a raw image) can be processed by image signal processing (e.g., an ISP utilizing an image signal processor) to create an image for a predetermined purpose. The ISP is typically performed by a series of pipeline stages implemented in hardware and / or software, and the sensor output processed by an existing ISP can be converted into an image that is ideal for human vision in various ways. The images created by the ISP can be used in applications designed for humans (such as photography), and can also be used in machine vision applications (such as autonomous driving). Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one general aspect, an image processing method includes: selecting a first image sensor from a pool of image sensors; generating a final image by processing an output image of the first image sensor using first sensor-specific image processing and using sensor-independent signal processing, the first sensor-specific image processing having a first individual setting specific to the first image sensor, and the sensor-independent signal processing having a setting common to image sensors in the pool of image sensors; and performing a first task based on the final image.

[0006] The first individual settings may be different from second individual settings dedicated to a second sensor-specific image processing of a second image sensor in the pool of image sensors.

[0007] The first individual setting and the second individual setting may differ in a specific image processing operation or in individual setting values ​​for the same specific image processing operation.

[0008] The image sensors in the image sensor pool may be configured to sense in different respective wavelength bands.

[0009] The different wavelength bands may include a visible light wavelength band and an infrared wavelength band.

[0010] The image sensor may include corresponding bandpass filters for different spectra positioned in different arrangements.

[0011] Each of the image sensors in the image sensor pool may be matched with a predetermined task based on sensor-specific signal processing, and the first image sensor is matched with a first task based on a first sensor-specific image processing.

[0012] A second image sensor in the pool of image sensors may be matched to the second task based on a second sensor-specific image processing having a second individual setting specific to the second image sensor.

[0013] The first image sensor may be matched to a third task based on a third sensor-specific image process having a third individual setting dedicated to the first image sensor.

[0014] The sensor-independent image processing may include sub-modes, and one of the sub-modes may be selected for the sensor-independent image processing according to environmental conditions.

[0015] Environmental conditions may include weather, location, illumination, or a combination thereof.

[0016] The first task may be a machine vision task including any one of object detection, object tracking, object classification, and segmentation, or any combination of any two or more thereof.

[0017] A non-transitory computer-readable storage medium may store any of the methods for performing the method.

[0018] In another general aspect, an electronic device includes: a sensor; one or more processors; and a memory storing instructions configured to cause the one or more processors to: select a first sensor from among the sensors; generate final data by processing output data output from the first sensor using first sensor-specific image processing and sensor-independent signal processing, wherein the first sensor-specific image processing has a first individual setting specific to the first sensor, and the sensor-independent signal processing has a setting common to the sensors; and perform a first task based on the final data.

[0019] The first individual setting may be different from a second individual setting dedicated to a second sensor-specific signal processing of the first sensor.

[0020] The sensors may be configured to sense signals in different respective wavelength bands.

[0021] Each of the sensors may be matched to a predetermined task based on sensor-specific signal processing, and a first sensor may be matched to a first task based on a first sensor-specific signal processing.

[0022] A second sensor among the sensors may be matched to the second task based on a second sensor-specific signal processing having a second individual setting specific to the second sensor.

[0023] The first sensor may be matched to a third task based on a third sensor-specific signal processing having a third individual setting dedicated to the first sensor.

[0024] The sensor-independent signal processing may include sub-modes, and one of the sub-modes may be selected for the sensor-independent signal processing according to environmental conditions.

[0025] Other features and aspects will be apparent from the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 An example of a machine task using a machine-perspective sensor and machine-perspective signal processing is shown in accordance with one or more embodiments.

[0027] Figure 2 Examples of sensor blocks, signal processing blocks, and task blocks are shown according to one or more embodiments.

[0028] Figure 3 An example of a task using a sub-mode of sensor-agnostic signal processing is shown in accordance with one or more embodiments.

[0029] Figure 4 An example of a machine view optimization process is shown in accordance with one or more embodiments.

[0030] Figure 5 An example of a process of configuring candidate sensors according to one or more embodiments is shown.

[0031] Figure 6 An example of a process for optimizing signal processing blocks and configuring a sensor pool in accordance with one or more embodiments is shown.

[0032] Figure 7An example of a machine perspective signal processing method according to one or more embodiments is shown.

[0033] Figure 8 An example of a configuration of an electronic device according to one or more embodiments is shown.

[0034] Throughout the drawings and detailed description, unless otherwise described or provided, it is understood that the same or similar reference numerals in the drawings represent the same or similar elements, features and structures. The drawings may not be to scale, and the relative sizes, proportions and depictions of the elements in the drawings may be exaggerated for clarity, illustration and convenience. DETAILED DESCRIPTION

[0035] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those sequences set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known after understanding the disclosure of the present application may be omitted.

[0036] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein that will be apparent after understanding the disclosure of the present application.

[0037] The terms used herein are only used to describe various examples and are not used to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. As used herein, the term "and / or" includes any one of the relevant listed items and any combination of any two or more. As non-limiting examples, the terms "comprise", "include", and "have" indicate that there are stated features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0038] Throughout the specification, when a component or element is described as being "connected to," "coupled to," or "engaged to" another component or element, it may be directly "connected to," "coupled to," or "engaged to" another component or element, or one or more other components or elements may reasonably exist between them. When a component or element is described as being "directly connected to," "directly coupled to," or "directly engaged to" another component or element, there may not be other elements between them. Similarly, expressions such as "between" and "immediately between," and "adjacent to" and "immediately adjacent to" may also be interpreted as described above.

[0039] Although terms such as "first," "second," and "third," or A, B, (a), (b), etc., may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions are not limited by these terms. Each of these terms is not used to define, for example, the nature, order, or sequence of the corresponding member, component, region, layer, or portion, but is only used to distinguish the corresponding member, component, region, layer, or portion from other members, components, regions, layers, or portions. Therefore, without departing from the teachings of the examples, the first member, first component, first region, first layer, or first portion referred to in the examples described herein may also be referred to as the second member, second component, second region, second layer, or second portion.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those of ordinary skill in the art to which the present disclosure belongs and the same meaning as those commonly understood based on the understanding of the disclosure of the present application. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) will be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the disclosure of the present application, and will not be interpreted in an idealized or overly formal sense. The use of the term "may" for an example or embodiment herein (e.g., with respect to what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto.

[0041] Figure 1 An example of a machine task using a machine-perspective sensor and machine-perspective signal processing according to one or more embodiments is shown. Figure 1, traditionally, a human-perspective sensor 110 and a human-perspective signal processing 120 are used to simulate or model the characteristics of human vision. The human-perspective sensor 110 and the human-perspective signal processing 120 can be used not only for applications designed for humans (such as photography), but also for machine tasks 130. For example, the human-perspective sensor 110 can be an image sensor based on a Bayer pattern, and the human-perspective signal processing 120 can be a pipeline of image signal processing (ISP) adjusted by humans. In short, the selection and / or configuration of the sensor and the selection and / or configuration of the ISP pipeline for processing the image of the sensor are usually oriented to the end use of humans (i.e., displaying images). However, when the machine task is the intended use of the sensed and processed images, this human-perspective selection and configuration method may not be optimal. That is, the properties of the image may not be optimal for the machine task. It may not be obvious which sensor selection / configuration and which pipeline selection / configuration is optimal for a given machine task.

[0042] As a non-limiting example, the machine task 130 may be a machine vision task, which includes any one of object detection, object tracking, object classification, and segmentation, or any combination of any two or more. The machine task 130 may be performed by a machine (e.g., images may be used for various computing tasks instead of human viewing). Since the image data provided to the machine task 130 by the human perspective sensor 110 and the human perspective signal processing 120 is generated from a human-centric perspective, when the machine task 130 is based on the human-centric image data, the capabilities of the machine may not be fully realized in the machine task 130.

[0043] The machine task 130 may be better accomplished when performed based on the machine perspective sensor 140 (e.g., selection / configuration of the sensor) and the machine perspective signal processing 150 (e.g., selection / configuration of the signal processing) rather than the human perspective sensor 110 and the human perspective signal processing 120. For example, instead of using a pipeline of an image sensor based on a Bayer pattern and an ISP tuned by humans (or tuned to the human visual system), a different sensor and a different signal processing selected / configured to improve the performance of the machine task 130 may be used as the machine perspective sensor 140 and the machine perspective signal processing 150. Techniques that emulate human cones (such as the Bayer pattern) may focus primarily on using visible light, and it may be difficult to know whether such techniques are optimal for any given machine vision task. For example, in a low light environment, it may be advantageous to use information from a white channel that uses the entire spectrum of visible light (without any color filters) to provide high light sensitivity. Furthermore, since predetermined colors tend to be used for many traffic signal / sign systems, lanes, or other critical information for autonomous driving, the number of pieces of driving-related information available (for example) may vary depending on which spectrum is used (i.e., which sensor and / or sensor settings / filters are used).

[0044] The scores of the machine task 130 can be measured for various sensors (e.g., which sensor and / or which sensor settings / filters) and various signal processing settings (e.g., which signal processing and / or which signal processing configuration settings). The machine perspective sensor 140 and machine perspective signal processing 150 with the highest score can be set as the sensor and signal processing settings for the machine task 130. For example, various color spectra can be obtained based on physical properties and processes. Using a generative (available) spectral pool as a search space, sensors with various filter arrays can be configured by repeated combinations of individual color pixels (of the sensor). Incidentally, the techniques described herein are not limited to sensors using visible light colors (e.g., infrared sensors or other sensors can be used). The machine perspective sensor 140 and machine perspective signal processing 150 can be selected / configured to maximize the performance of the machine task 130.

[0045] Figure 2 Examples of sensor blocks, signal processing blocks, and task blocks are shown according to one or more embodiments. Figure 2, any of the tasks of the task block 230 (e.g., the first task 231, the second task 232, and the third task 233) can potentially be performed based on: (i) any of a set of sensors (e.g., the first sensor 211, the second sensor 212, and the third sensor 213) of the sensor block 210, (ii) any of the types of signal processing (e.g., sensor-specific signal processing 221) of the signal processing block 220, and (iii) sensor-independent signal processing 226 of the signal processing block 220.

[0046] For example, for the first task 231, (i) the first sensor 211 may be selected from the group of sensors of the sensor block 210, (ii) signal processing of image data from the first sensor 211 may be performed by selecting and using a first sensor-specific signal processing 222 having a first individual setting dedicated to (specific to) the first sensor 211, and (iii) a sensor-independent signal processing 226 (having a common setting for multiple sensors of the sensor block 210); the first task 231 may be performed based on the signal / image results.

[0047] The sensors in the set of sensors of the sensor block 210 may each sense electromagnetic (light) energy in a respective different wavelength band (which may overlap). For example, the different wavelength bands may include two or more of a visible light wavelength band (e.g., a red wavelength band, a green wavelength band, a blue wavelength band, or a combination thereof), an infrared wavelength band (e.g., a near infrared (NIR) wavelength band), a wavelength band corresponding to a radio signal, and a wavelength band corresponding to a light wave signal, etc. For example, the set of sensors may include a sensor configured to capture a visible light signal, a sensor configured to capture an infrared signal, a sensor configured to capture (e.g., simultaneously) a visible light signal and an infrared signal, a radar configured to use a radio signal, a LiDAR configured to use a light wave signal, etc.

[0048] The set of sensors of the sensor block 210 may include sensors including bandpass filters of different spectra positioned in different arrangements. The bandpass filters may be arranged in a pixel array (pixel filter) such as a color filter array (CFA). For example, pixels corresponding to bandpass filters that transmit electromagnetic waves of white (W), red (R), green (G), cyan (C), magenta (M), or NIR wavelengths may be arranged in any of a variety of repeating patterns such as 1×1, 2×2, 3×3, and 4×4 within the CFA.

[0049] Each signal processing in the sensor-specific signal processing 221 may have individual settings dedicated to a corresponding predetermined sensor (in the sensor block 210); each sensor may have more than one signal processing. The sensor-independent signal processing 226 may have one or more common settings for all sensors (settings applicable to all sensors). For example, when the ISP pipeline is configured by the signal processing block 220, the sensor-specific stages of the ISP pipeline may be configured according to the sensor-specific signal processing, and the stages of the ISP pipeline applicable to all sensors in the sensor block 210 may be configured according to the sensor-independent signal processing 226.

[0050] Types of sensor-specific signal processing (e.g., first sensor-specific signal processing 222, second sensor-specific signal processing 223, and third sensor-specific signal processing 224) may have individual settings that are different from each other. For example, the individual settings of the first sensor-specific signal processing 222 may be different from the individual settings of the second sensor-specific signal processing 223 (although any two sensors may have the same sensor-specific settings, their settings will generally be different). The individual settings may specify different detailed signal processing operations (e.g., which pipeline stages are active) and / or different individual setting values ​​(e.g., of the stages). For example, the first sensor-specific signal processing 222 may include more detailed signal processing operations than the second sensor-specific signal processing 223. For example, the detailed signal processing operations may configure the ISP pipeline to include and / or have settings for stages such as demosaicing, denoising, automatic white balance (AWB), and color correction matrix (CCM). For example, even when the first sensor specific signal processing 222 and the second sensor specific signal processing 223 include the same signal processing operation / level, the individual setting value of the operation / level in the first sensor specific signal processing 222 may be different from the individual setting value of the operation / level in the second sensor specific signal processing 223. For example, a parameter value for performing demosaicing, denoising, AWB, CCM, etc. may be set to one value in the detailed signal processing operation of the first sensor specific signal processing 222, and set to another value in the detailed signal processing operation of the second sensor specific signal processing 223.

[0051] The tasks of task block 230 (e.g., first task 231, second task 232, and third task 233) may correspond to corresponding machine tasks. For example, each machine task may be a type of task performed by a machine to simulate human activities (e.g., using machine learning techniques). For example, a machine task may include a machine vision task that simulates vision-based human activities. Machine vision tasks may include object detection, object tracking, object classification, and segmentation, to name some examples. Thus, for example, first task 231, second task 232, and third task 233 may be object detection, object tracking, and object classification, respectively. Any of a variety of applications (such as, autonomous driving) may be performed based on machine vision tasks.

[0052] Each of the tasks in the task block 230 can be matched with both the task-optimal individual sensor of the sensor block 210 and the task-optimal individual signal processing of the sensor-specific signal processing 221. These matches can be performed in advance by an optimization process to match each task with the sensor and sensor-specific signal processing that are optimal for the task. In other words, the combination of the sensor and the sensor-specific signal processing that maximizes the performance of the corresponding task can be matched with the task. For example, through the optimization process, the first task 231 can be matched with the first sensor 211 and the first sensor-specific signal processing 222; the first sensor 211 and the first sensor-specific signal processing 222 have been found to be the optimal combination of the first task 231. Similarly, the second task 232 can be matched with the second sensor 212 and the second sensor-specific signal processing 223; the second sensor 212 and the second sensor-specific signal processing 223 have been found to be the optimal combination of the second task 232. The same sensor can be matched with different tasks, but with different sensor-specific signal processing combinations. For example, the first sensor 211 can be matched with the first task 231 together with the first sensor-specific signal processing 222. Meanwhile, the first sensor 211 may be matched with the third task 233 together with the third sensor specific signal processing 224 .

[0053] Figure 3 An example of a task using a sub-mode of sensor-independent signal processing according to one or more embodiments is shown. Figure 3 , and reference Figure 2Different from the described examples, the sensor-independent signal processing 326 may include sub-modes (e.g., a first sub-mode 327, a second sub-mode 328, and a third sub-mode 329). One of the sub-modes may be selected for the sensor-independent signal processing 326 according to the environmental conditions 301. For example, the environmental conditions 301 may include at least one of weather, location, illumination, and a type of surrounding area. The indicated weather may include clear weather, snowy weather, rainy weather, foggy weather, etc. The location may include a tunnel, an urban area, a suburb, a city, a country, etc. The illumination may include high illumination and low illumination. The type of surrounding area may include a desert, a grassland, a forest, a mountain, a plain, etc. The environmental conditions 301 may be obtained by a sensor of a sensor block, a sensor other than a sensor of a sensor block, a network, or a combination thereof.

[0054] For example, when weather is used as environmental condition 301, one of the sub-modes may be selected according to the weather. For example, when the weather is sunny, first sub-mode 327 may be used for sensor-independent signal processing 326, when the weather is raining or snowing, second sub-mode 328 may be used for sensor-independent signal processing 326, and when the weather is foggy, third sub-mode 329 may be used for sensor-independent signal processing 326. The sub-modes may have different individual settings that are independent of the sensor and related to environmental condition 301. When the weather is indicated as sunny, first sub-mode 327 may be used for sensor-independent signal processing 326 regardless of which sensor or sensor-specific signal processing is used for a given task.

[0055] Figure 4 An example of a machine perspective optimization process according to one or more embodiments is shown. Figure 4 , a sensor pool may be obtained from the candidate sensors 410 through the optimization model 450, and a sensor-specific signal processing 421 and a sensor-independent signal processing 422 may be determined for the sensor pool. For example, detailed signal processing operations (e.g., which pipeline stage to use) and individual setting values ​​(e.g., of the stage) of the sensor-specific signal processing 421 and the sensor-independent signal processing 422 may be determined.

[0056] The optimization model 450 may select any candidate sensor from among the candidate sensors 410. Sensor-specific signal processing 421 and sensor-independent signal processing 422 may be performed on image data (e.g., raw images) from the corresponding candidate sensor. The image processing result may be an output image 430. Each of the machine vision tasks 440 may be performed based on the output image 430. The machine vision tasks 440 may include any one of object detection, object tracking, object classification, and segmentation, or any combination of any two or more. The output image 430 and the machine vision tasks 440 are non-limiting examples of machine tasks; optimization may be performed on other machine vision tasks.

[0057] The optimization model 450 may optimize the sensor-specific signal processing 421 and the sensor-independent signal processing 422 based on the score of the machine vision task 440. For example, the detailed signal processing operations (e.g., pipeline stages) and individual setting values ​​(e.g., settings of stages) of the sensor-specific signal processing 421 and the sensor-independent signal processing 422 may be optimized. The optimization model 450 may repeatedly perform the optimization operation for the candidate sensors 410. As a result, a sensor pool may be obtained from the candidate sensors 410, and the sensor-specific signal processing 421 and the sensor-independent signal processing 422 for the sensor pool may be determined.

[0058] To reiterate, for each machine vision task 440, different combinations of sensors, sensor-specific signal processing, and sensor-independent signal processing may be used to generate a corresponding output image, and the output image may be scored for suitability (optimality) for the corresponding machine vision task 440. For each machine vision task, the optimal (highest scoring) combination of sensors, sensor-specific signal processing, and sensor-independent signal processing may be set as the optimal combination for the corresponding machine vision task.

[0059] Figure 5 An example of a process for configuring a candidate sensor (eg, candidate sensor 410) according to one or more embodiments is shown. Figure 5 , the spectrum pool 511 may include labels of spectrums in various frequency bands (such as W, R, G, blue (B), C, M, and NIR). The array pool 512 may include labels of various batch patterns (such as 1×1, 2×2, 3×3, and 4×4). The candidate sensor 520 may be obtained by combining the sensor attributes 510 in various ways. For example, a sensor using a pattern of RGGB repeatedly arranged in a 2×2 grid, a sensor using a pattern of CMYW repeatedly arranged in a 2×2 grid, a sensor using a pattern of W repeatedly arranged in a 1×1 grid, etc. may be obtained as the candidate sensor 520. Many candidate sensors (e.g., candidate sensor 520) may be obtained in the optimization process, and the sensor pool may be configured with the candidate sensor 520.

[0060] Figure 6 An example of a process for optimizing a signal processing block (eg, signal processing block 220) and configuring a sensor pool (eg, sensor block 210) according to one or more embodiments is shown. Figure 6, the sensor-independent signal processing may be fixed in operation 610, and a sensor may be selected from among the candidate sensors in operation 620. Initially, the sensor-independent signal processing may be set based on a known signal processing (e.g., human view signal processing). In operation 630, the sensor-specific signal processing of the selected sensor may be trained so that the score of the given task is maximized. The training may be performed by iteratively updating the detailed signal / image processing operations and individual setting values ​​of the sensor-specific signal processing.

[0061] In operation 640, it is checked whether the training of all candidate sensors is completed. When there is a candidate sensor for which training has not been performed, operation 620 and / or operation 630 may be performed again on another candidate sensor. When the training of all candidate sensors is completed, a sensor pool may be configured in operation 650. The sensor pool may be configured with sensors that excel in a given task among the candidate sensors. In operation 660, using the sensors of the sensor pool, sensor-specific signal processing and sensor-independent signal processing may be adjusted (e.g., accurately adjusted). In operation 660, unlike in operation 610, sensor-independent signal processing may be changed.

[0062] Figure 7 An example of a machine perspective signal processing method according to one or more embodiments is shown. Figure 7 The electronic device may select a first sensor from among available sensors in operation 710, process output data (e.g., raw image data) of the first sensor using a first sensor-specific signal processing having a first individual setting dedicated to the first sensor and a sensor-independent signal processing having a common setting for a plurality of sensors in operation 720, and perform a first task based on the processed output data in operation 730.

[0063] The first individual setting may be different from a second individual setting dedicated to a second sensor-specific signal processing of a second sensor of the plurality of sensors.

[0064] The first individual setting and the second individual setting may differ in one or more of a detailed signal processing operation and an individual setting value.

[0065] The sensor can sense signals in different wavelength bands.

[0066] The different wavelength bands may include a visible wavelength band and an infrared wavelength band.

[0067] The sensor may include sensors having different spectral bandpass filters arranged in different ways.

[0068] Each of the sensors may be matched with a predetermined task based on sensor-specific signal processing, and a first sensor may be matched with a first task based on a first sensor-specific signal processing.

[0069] A second sensor of the plurality of sensors may be matched to the second task based on a second sensor-specific signal processing having a second individual setting specific to the second sensor.

[0070] The first sensor may be matched to a third task based on a third sensor-specific signal processing having a third individual setting dedicated to the first sensor.

[0071] The sensor-independent signal processing may include sub-modes, and one of the sub-modes may be selected for the sensor-independent signal processing according to environmental conditions.

[0072] Environmental conditions may include weather, location, illumination, or a combination thereof.

[0073] The first task may be a machine vision task, which includes one or more of detection, tracking, classification, and segmentation.

[0074] In addition, refer to Figures 1 to 6 and Figure 8 The description provided can be applied to Figure 7 signal processing method.

[0075] Figure 8 An example of a configuration of an electronic device according to one or more embodiments is shown. Figure 8 , the electronic device 800 may include a processor 810, a memory 820, a sensor block 830, a storage device 840, an input device 850, an output device 860, and a network interface 870. The processor 810, the memory 820, the sensor block 830, the storage device 840, the input device 850, the output device 860, and the network interface 870 may communicate with each other via a communication bus 880. For example, the electronic device 800 may be implemented as at least a part of a mobile device (such as a mobile phone, a smart phone, a personal digital assistant (PDA), a netbook, a tablet computer, or a laptop computer), a wearable device (such as a smart watch, a smart band, or smart glasses), a computing device (such as a desktop or a server), a home appliance (such as a TV, a smart TV, or a refrigerator), a security device (such as a door lock), or a vehicle (such as an autonomous vehicle or a smart vehicle).

[0076] The processor 810 may execute instructions and functions in the electronic device 800. For example, the processor 810 may process instructions stored in the memory 820 or the storage device 840. The processor 810 may execute instructions and functions in the electronic device 800. Figures 1 to 7When the functions and instructions are executed by the processor 810, the electronic device 800 may be controlled accordingly. For example, when the instructions are executed by the processor 810, the instructions may cause the electronic device 800 to select a first sensor from among a plurality of sensors, process output data of the first sensor using a first sensor-specific signal processing having a first individual setting dedicated to the first sensor and a sensor-independent signal processing having a common setting for a plurality of sensors, and perform a first task based on the processed output data.

[0077] The memory 820 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The memory 820 may store instructions to be executed by the processor 810 and store information associated with the software and / or application when the software and / or application is being executed by the electronic device 800.

[0078] The sensor block 830 may include a plurality of sensors. For example, the plurality of sensors may include a sensor for capturing visible light signals, a sensor for capturing infrared signals, a sensor for capturing visible light signals and infrared signals, a radar using radio wave signals, a LiDAR using light wave signals, and the like.

[0079] Storage device 840 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. Storage device 840 may store a larger amount of information than memory 820 and may store information for a longer period of time. For example, storage device 840 may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other types of non-volatile memory known in the art.

[0080] The input device 850 may receive input from the user through a traditional input scheme using a keyboard and a mouse and through a new input scheme (such as touch input, voice input, or image input). The input device 850 may include, for example, a keyboard, a mouse, a touch screen, a microphone, or any other device that can detect input from the user and send the detected input to the electronic device 800. The output device 860 may provide the output of the electronic device 800 to the user through a visual, auditory, or tactile channel. The output device 860 may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that can provide output to the user. The network interface 870 may communicate with an external device through a wired or wireless network.

[0081] Here about Figures 1 to 8The computing devices, electronic devices, processors, memories, image sensors, displays, information output systems and hardware, storage devices and other devices, devices, units, modules and components described are implemented by hardware components or represent hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. A hardware component implemented by a processor or a computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware component can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component, or two or more hardware components. The hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0082] Perform the operations described in this application Figures 1 to 8The method shown in is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as executing instructions or software as described above to implement the operations performed by the method described in this application. For example, a single operation, or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation, or two or more operations.

[0083] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions herein, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0084] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), programmable random access read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-Re, Blu-ray or optical disk storage device, hard disk drive (HDD), solid state drive (SSD), flash memory, card memory (such as, MultiMediaCard Micro or card (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner, and provide the instructions or software and any associated data, data files and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed by one or more processors or computers in a distributed manner.

[0085] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered to be descriptive only, not for limiting purposes. The description of the features or aspects in each example should be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described system, architecture, device, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved.

[0086] Therefore, in addition to the above disclosure, the scope of the disclosure may be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.

Claims

1. An image processing method, comprising: selecting a first image sensor from a pool of image sensors; generating a final image by processing an output image of an image sensor using first sensor-specific image processing having first individual settings specific to the first image sensor and using sensor-independent signal processing having settings common to image sensors in a pool of image sensors; and A first task is performed based on the final image.

2. The image processing method according to claim 1, wherein: The first individual settings are different from second individual settings dedicated to a second sensor-specific image processing of a second image sensor in the pool of image sensors.

3. The image processing method according to claim 2, wherein: The first individual setting and the second individual setting differ in a specific image processing operation or in individual setting values ​​of the same specific image processing operation.

4. The image processing method according to claim 1, wherein: The image sensors in the image sensor pool are configured to sense in different respective wavelength bands.

5. The image processing method according to claim 4, wherein: The different corresponding wavelength bands include two or more of a visible light wavelength band, an infrared wavelength band, a wavelength band corresponding to a radio signal, and a wavelength band corresponding to an optical wave signal.

6. The image processing method according to claim 1, wherein: The image sensor includes corresponding bandpass filters for different spectra positioned in different arrangements.

7. The image processing method according to claim 1, wherein Each of the image sensors in the image sensor pool is matched to a predetermined task based on sensor specific signal processing, and The first image sensor is matched to the first task based on a first sensor specific image processing.

8. The image processing method according to claim 7, wherein: A second image sensor in the pool of image sensors is matched to the second task based on a second sensor-specific image processing having a second individual setting specific to the second image sensor.

9. The image processing method according to claim 7, wherein: The first image sensor is matched to the third task based on a third sensor-specific image processing having a third individual setting dedicated to the first image sensor.

10. The image processing method according to any one of claims 1 to 9, wherein Sensor-independent image processing includes sub-modes, and One of the sub-modes is selected for sensor-independent image processing depending on environmental conditions.

11. The image processing method according to claim 10, wherein: The environmental conditions include at least one of weather, location, illumination, and type of surrounding area.

12. The image processing method according to any one of claims 1 to 9, wherein: The first task is a machine vision task, which includes any one of object detection, object tracking, object classification and segmentation, or any combination of any two or more thereof. 13 . A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the image processing method according to claim 1 .

14. An electronic device, comprising: Multiple sensors; one or more processors; as well as A memory storing instructions configured to cause the one or more processors to: selecting a first sensor from among the plurality of sensors; generating final data by processing output data output from the first sensor using first sensor-specific image processing having a first individual setting dedicated to the first sensor and sensor-independent signal processing having a setting common to the plurality of sensors; and The first task is performed based on the final data.

15. The electronic device according to claim 14, wherein: The first individual setting is different from a second individual setting dedicated to a second sensor-specific signal processing of a second sensor.

16. The electronic device according to claim 14, wherein: The plurality of sensors are configured to sense signals in different respective wavelength bands.

17. The electronic device according to claim 14, wherein Each of the plurality of sensors is matched to a predetermined task based on sensor specific signal processing, and The first sensor is matched to the first task based on the first sensor specific signal processing.

18. The electronic device according to claim 17, wherein: A second sensor among the plurality of sensors is matched to the second task based on a second sensor-specific signal processing having a second individual setting dedicated to the second sensor.

19. The electronic device according to claim 17, wherein: The first sensor is matched to the third task based on a third sensor-specific signal processing having a third individual setting dedicated to the first sensor.

20. The electronic device according to any one of claims 14 to 19, wherein Sensor-independent signal processing includes sub-modes, and One of the sub-modes is selected for sensor-independent signal processing based on environmental conditions.

Images