Data-driven intelligent sensor design approach for machine vision

Abstract

This invention relates to a data-driven intelligent sensor design method for machine vision. The resulting ISP configuration strategy is purely data-driven, yielding different configuration strategies based on different inputs and tasks. A hybrid reinforcement learning and supervised learning approach is employed for optimization, resolving the non-differentiability issue of the strategy. Reconfiguration is performed based on existing ISP links without altering the existing algorithm structure, ensuring compatibility with existing ISP links and ease of hardware implementation. The number of algorithms can be further expanded based on existing algorithm links, while the structure of the policy network does not require significant changes; only the number of output strategies needs modification. This significantly reduces sensor power consumption.

Description

Technical Field

[0001] This invention relates to an artificial intelligence technology, and more particularly to a data-driven intelligent sensor design method for machine vision. Background Technology

[0002] Computer vision studies how to enable computers to understand images, and a typical computer vision system consists of a front-end imaging system and a back-end vision system. Currently, most computer vision algorithms use pre-processed color images as input, and these images are almost always captured by cameras equipped with single-sensor solutions. This solution covers the sensor with a color filter array, and to obtain a color image that conforms to human vision, the captured image needs to undergo a series of processing steps; these algorithms are called the Image Signal Processing pipeline (ISP pipeline). However, ISPs are designed for the human eye, and the raw images captured by the sensor already contain all scene-related information. Excessive processing may actually cause information loss and make pixel values ​​that are linearly related to the scene become non-linear.

[0003] Furthermore, the ISP module in the sensor also introduces additional power consumption and area requirements to the vision system. Studies have shown that a typical ISP module consumes 250mW when processing 1920×1080 resolution video at 60 frames per second, while a dedicated detection chip processing 1920×1080 video in real time consumes only 58.6mW, approximately 23% of the ISP module's power consumption. This demonstrates that a dedicated ISP design can significantly reduce its power consumption.

[0004] However, in existing computer vision systems, the front-end imaging system and the back-end vision system are designed and optimized separately, and the optimization targets are different. The former is designed to adapt to the human eye, while the latter is designed to enable computers to better understand the computer.

[0005] Therefore, in order to ensure the accuracy of computer vision tasks while minimizing the power consumption of the imaging component, it is necessary to jointly optimize the front-end imaging and back-end tasks. The goal of optimization is to find an image processing link configuration strategy that can guarantee or even improve task accuracy for different inputs while using only the minimum number of ISP algorithms, and to guide the design of smart sensors. Summary of the Invention

[0006] To address the design challenge of image processing links in computer vision systems that consider both the front-end imaging system and the back-end vision system, a data-driven intelligent sensor design method for machine vision is proposed.

[0007] The technical solution of the present invention is as follows: a data-driven intelligent sensor design method for machine vision, which constructs a policy network between the imaging system and the computer vision system. The policy network obtains the image processing policy of the image output by the imaging system and optimizes it in association with the task in the vision system. The image processing policy is then fed back to the imaging system to reconstruct the image processing link.

[0008] Furthermore, the policy network includes four residual blocks and a fully connected layer. The length of the fully connected layer is set according to the number N of imaging processing algorithms in the image processing strategy of the imaging system. The fully connected layer outputs a recognition vector for the image processing strategy used by the imaging system.

[0009] Furthermore, the optimization method involves constructing a reward function based on the image processing strategy of the input image and the consistency between task execution and label in the corresponding image vision system. The optimization objective is to achieve the task in the vision system while minimizing the amount of image processing.

[0010] Furthermore, the image processing strategy method of the reconstructed imaging system is as follows:

[0011] The image processing strategy for the imaging system output image identified by the policy network is an N-dimensional vector, represented as:

[0012] f isp (x,θ p )=[g1,g2,...,g N ]

[0013] Among them, f isp (x,θ p ) is a parameter of θ p Image processing strategy, where x is the input image, g i ∈[0,1] represents the output after sigmoid processing, indicating that a i Given a probability of 1, the output distribution of the image processing strategy is expressed as:

[0014] The output policy identified by the policy network is:

[0015]

[0016] Let the series of policies identified by the policy network be represented as follows:

[0017]

[0018] in The parameter is θ p The policy network takes actions based on the input image x, and the loss function of the entire framework can be expressed as:

[0019]

[0020] in, The parameter is θ t The loss function for the visual task is f(x), where f(x) is the image after the image processing chain is reconstructed, and y is the label corresponding to the visual task.

[0021] Then, during the optimization process, the gradient of the loss function can be calculated as follows:

[0022]

[0023] R(A) is the reward function. To encourage the policy network to explore more possibilities, the output entropy of the policy network is added to the loss function, which can be written as:

[0024]

[0025] Where α is the weight of this term in the overall loss function.

[0026] A method for reconstructing the image processing link in an imaging system of a computer vision system is proposed. This method optimizes the image processing link in the front-end imaging system by associating it with the specific tasks in the back-end vision system, thereby implementing a minimum amount of image processing algorithm for tasks in the vision system and reconstructing the image processing link.

[0027] The beneficial effects of this invention are as follows: The data-driven intelligent sensor design method for machine vision obtained in this invention produces a purely data-driven ISP configuration strategy, which can obtain different configuration strategies based on different inputs and tasks. It employs a hybrid reinforcement learning and supervised learning method for optimization, solving the problem of non-differentiability of the strategy. It reconfigures the ISP based on existing links without changing the existing algorithm structure, ensuring compatibility with existing ISP links and ease of hardware implementation. Based on existing algorithm links, the number of algorithms can be further expanded, while the structure of the policy network does not need to be significantly changed; only the number of output strategies needs to be modified. This can significantly reduce sensor power consumption. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the data-driven intelligent sensor design method for machine vision according to the present invention.

[0029] Figure 2 A detailed structural diagram of the entire framework for implementing the method of this invention;

[0030] Figure 3 This is a detailed structural diagram of the strategy network part of the present invention;

[0031] Figure 4 The present invention provides an example diagram of the minimum computer vision system design architecture for different tasks under this strategy. Detailed Implementation

[0032] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0033] like Figure 1 The diagram illustrates a data-driven approach to designing intelligent sensors for machine vision. An input image is first fed into an ISP Policy Network (ISPNet). After passing through the ISPNet, the corresponding ISP link policy is obtained. This policy, along with the image, is then input into a reconfigurable ISPNet link / ISP configuration to obtain a processed image. The processed image is then fed into backend computer vision task algorithms.

[0034] Figure 2 The main structure of the entire framework is presented. The image is first fed into a policy network consisting of four ResNet blocks to derive the ISP link configuration policy for the input image. These policies are vectors composed of {0, 1}. In this example, each ISP link includes 10 steps sequentially, so the output vector after passing through the policy network has a length of 10. In the vector, 1 represents that the step is executed, and 0 represents that the step is skipped. From the output vector, we can see the policy executed by the input image in the front-end imaging system. For example, the following are nine image processing algorithms included in the imaging system's imaging processing link, with nine steps connected sequentially in the imaging processing link, as follows:

[0035] 1) Bad Pixel Correction: Corrects dark spots, glare, and other abnormal pixels in RAW images;

[0036] 2) Anti-aliasing filtering: reduces artifacts in subsequent processing;

[0037] 3) Bayer domain noise reduction: Denoising the image in the Bayer domain;

[0038] 4) De-mosaic: Interpolate the single-channel Bayer image to a three-channel RGB image;

[0039] 5) White Balance: Resolves color cast issues in images by calibrating the whites in the image;

[0040] 6) Gamma correction: Compresses the dynamic range of an image to meet display requirements while maintaining the image's visual characteristics;

[0041] 7) Sharpening: Enhances the edges of objects in an image;

[0042] 8) Histogram equalization: Adjusts image contrast by equalizing the pixel distribution of the image;

[0043] 9) RGB domain noise reduction: Reduce noise in the RGB domain of the image.

[0044] Here, because the policy network's processing policy after feature extraction from the image acquired by the imaging system outputs a binary hard decision vector of 0 or 1, and such an output is non-differentiable for optimization methods based on continuous gradients, this invention uses association learning to solve this problem. Since the output of ISPNet consists of N independent {0,1} values, it resembles an N-fold Bernoulli distribution. Therefore, the output of ISPNet is an N-dimensional vector, which can be represented as: f isp (x,θ p )=[g1,g2,...,g N ]

[0045] Among them, f isp (x,θ p ) is a parameter of θ p Image processing strategy, where x is the input image, g i ∈[0,1] represents the output after sigmoid processing, indicating that a i The probability is 1. Therefore, the output distribution of ISPNet is expressed as:

[0046]

[0047] Therefore, the output strategy of ISPNet is:

[0048]

[0049] Here, π(A|x,θ) p ) represents the policy function corresponding to ISPNet, and the decision value is a. i ∈{0,1} is based on g i Sampled, a i =1 means the i-th step in the ISP should be executed, while 0 means the step should be skipped. If f is used... s_i Let i represent the i-th step in the ISP link, then its output can be represented as:

[0050] μ i+1 =a i f s_i (μ i )+(1-a i )μ i

[0051] Where, μ i This is the input at step i in the ISP link; correspondingly, μ1 = x is the original RAW image input. To encourage skipping as many steps in the ISP link as possible, the output step of ISPNet uses the following reward function:

[0052]

[0053] Where w i ∈[0,1] represents the weight corresponding to the i-th step in the ISP link execution. This weight can be measured by metrics such as algorithm complexity.

[0054] The processed images, according to the strategy, are then input into ISPnet for corresponding visual tasks, such as... Figure 2 The visual tasks listed include image classification, object detection, image enhancement, etc. To simultaneously skip as many ISP steps as possible while maintaining accuracy in the visual tasks, the results obtained from the visual tasks need to be optimized in conjunction with ISPNet. To achieve this, a reward factor λ is added to evaluate the performance of the visual tasks. r and penalty factor λ p The reward function R(A) output by ISPNet is rewritten as:

[0055]

[0056] Taking classification tasks as an example, when the classification output matches the label, a positive factor λ is obtained. r =λ, otherwise a penalty factor λ is obtained. p =-λ.

[0057] By correlating the rewards of the policy network with the performance of the vision task, the entire task can be optimized together. The goal of training the reconstructed ISP link policy network is to maximize the reward, and this series of policies can be represented as:

[0058]

[0059] in The parameter is θ p The policy network takes actions based on the input image x. Therefore, the loss function of the entire framework can be expressed as:

[0060]

[0061] in, The parameter is θ t The loss function for the visual task is f(x), where f(x) is the image after the image processing chain is reconstructed, and y is the label corresponding to the visual task.

[0062] Then, during the optimization process, the gradient of the loss function can be calculated as follows:

[0063]

[0064] Furthermore, to encourage the policy network to explore more possibilities, the output entropy of the policy network is added to the loss function, which can be written as:

[0065]

[0066] Where α is the weight of this term in the overall loss function.

[0067] Figure 3 The detailed structure of the policy network is presented, consisting of four residual blocks and one fully connected layer. The length of the fully connected layer is set according to the number of image processing algorithms in the ISP link of the imaging system. This makes the overall framework more flexible; when it is necessary to add or remove algorithms in the ISP link, only the number of outputs of the fully connected layer needs to be changed to make the entire framework run and obtain the corresponding output policy.

[0068] Figure 4 This paper presents minimal computer vision system design architecture examples for different tasks under this strategy. For classification tasks, using the above framework, the final result is that all steps in the ISP can be skipped, and subsequent tasks can directly use the RAW images output by the image sensor, thereby saving the energy consumption of the entire system. For object detection tasks, the minimum configuration is that the demosaicing step needs to be retained in the ISP, while other steps can be skipped.

[0069] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A data-driven intelligent sensor design method for machine vision, characterized in that, A policy network is constructed between the imaging system and the computer vision system. The policy network obtains the image processing policy of the image output by the imaging system and optimizes it in association with the task in the vision system. The image processing policy is then fed back to the imaging system to reconstruct the image processing link. The policy network includes four residual blocks and one fully connected layer. The length of the fully connected layer is set according to the number N of imaging processing algorithms in the image processing strategy of the imaging system. The fully connected layer outputs the recognition vector of the image processing strategy used by the imaging system. The image processing strategy for reconstructing the imaging system is as follows: Since the processing strategy obtained by the policy network after feature extraction of the image acquired by the imaging system outputs a binary hard decision vector of 0 or 1, such an output is non-differentiable for optimization methods based on continuous gradients. Because the output of the policy network consists of N independent {0,1} values, the image processing strategy for the imaging system output image identified by the policy network is an N-dimensional vector, represented as: f isp (x,θ p )=[g1,g2,...,g N ] Among them, f isp (x,θ p ) is a parameter of θ p Image processing strategy, where x is the input image, g i ∈[0,1] represents the output after sigmoid processing, indicating that a i Given a probability of 1, the output distribution of the image processing strategy is expressed as: The output policy of the policy network is: Where π(A|x,θ) p Let be the policy function corresponding to the policy network, and let a be the decision value. i ∈{0,1} is based on g i Sampled, a i =1 means the i-th step in the policy network should be executed, while 0 means the step should be skipped; if f is used s_i If we denote the i-th step in the policy network link, then its output is represented as: m i+1 =a i f s_i (m i )+(1-a i )m i Where, μ i It is the input of the i-th step in the ISP link, and correspondingly, μ1 = x is the original RAW image input; To encourage skipping as many steps as possible in the policy network links, the output steps of the policy network use the following reward function: Where w i ∈[0,1] represents the weight corresponding to the i-th step in the execution strategy network link, and this weight is measured according to the algorithm complexity index; The image processed according to the policy is then input into the policy network to perform the corresponding visual task. In order to skip as many policy network steps as possible while ensuring the accuracy of the visual task, the result of the visual task needs to be optimized in conjunction with the policy network, and the reward factor λ for judging the effect of the visual task needs to be increased. r and penalty factor λ p The reward function R(A) output by the policy network is rewritten as: By correlating the rewards of the policy network with the performance of the vision task, the entire task can be optimized together.

2. The data-driven intelligent sensor design method for machine vision according to claim 1, characterized in that, The optimization method is as follows: a reward function is constructed based on the image processing strategy of the input image and the consistency between task execution and label in the corresponding image vision system. The optimization objective is to complete the task in the vision system while minimizing the amount of image processing algorithm.

Images