Visual sensor sampling parameter adaptive learning system

Abstract

The application discloses a visual sensor sampling parameter self-adaptive learning system, which comprises a visual sensor; a standard sample image for optical testing is acquired, visual sensor sampling parameter self-adaptive learning is carried out, and an algorithm model for the category of visual sensor is obtained; the algorithm model uses the current sampling parameter of the visual sensor to sample and obtain image features, and automatically adjusts the sampling parameter of the visual sensor through the algorithm model, so that sampling is synchronized with screen refreshing action; the visual sensor and the algorithm model are combined and used for attribute measurement and calculation of other screens; in the measurement process, the algorithm model automatically adjusts the sampling parameter of the visual sensor, and eliminates the stroboscopic texture noise in the sampling image; the application supports adaptive learning of different display screen categories, different visual sensor categories and different frequency ranges, and automatically learns with scene changes, and has high robustness.

Description

Technical Field

[0001] This invention relates to the field of adaptive learning, and more specifically, to an adaptive learning system for visual sensor sampling parameters. Background Technology

[0002] The measurement of optical properties of a display includes its time stability, brightness and color uniformity, color gamut, chromaticity constancy, channel independence, and color temperature. Traditional display measurements require displaying a pure color image of a specific color on the screen, and simultaneously using multiple devices such as luminance meters, colorimeters, and vision sensors to measure the screen's optical characteristics and compare them with known sample image features to calculate these optical properties. This type of equipment is diverse, expensive, requires stringent ambient lighting conditions, and is large and heavy, typically needing to be deployed in a specialized optical laboratory, presenting the following problems:

[0003] 1. It is impossible to measure the frequency of a monitor playing arbitrary video frames because photoelectric sensors measure the absolute change in the brightness of the monitor. However, screen refresh and switching of screen content will cause differences in brightness between different moments. Existing methods all measure the solid color screen to obtain the waveform of the screen brightness change over time, thereby obtaining the monitor refresh rate.

[0004] 2. It is impossible to track the frequency of displays with dynamically changing refresh rates. For example, VR glasses have a low-power mode for rendering images, which requires dynamically adjusting the display's operating parameters based on changes in the image content and the VR headset's posture. When measuring its photoelectric properties, if the frequency of subsequent moments can be predicted based on the frequency changes over a period of time, a shorter response time will result in better dynamic performance measurement. However, photoelectric sensors only measure the instantaneous frequency and cannot correlate or predict the frequency change trend over a previous period of time. Summary of the Invention

[0005] To address at least one of the aforementioned technical problems, this invention proposes an adaptive learning system for visual sensor sampling parameters.

[0006] The first aspect of the present invention provides an adaptive learning system for sampling parameters of a visual sensor, comprising: a visual sensor;

[0007] Acquire standard sample images for optical testing, perform adaptive learning of visual sensor sampling parameters, and obtain an algorithm model for this type of visual sensor;

[0008] The algorithm model uses the current sampling parameters of the visual sensor to obtain image features, and automatically adjusts the sampling parameters of the visual sensor to achieve synchronous sampling with the screen refresh action.

[0009] By combining visual sensors and algorithm models, the properties of other screens can be measured and calculated. During the measurement process, the algorithm model automatically adjusts the sampling parameters of the visual sensors to eliminate flicker texture noise in the sampled images.

[0010] In a preferred embodiment of the present invention, the standard sample image includes five standard colors: red, green, blue, black, and white.

[0011] In a preferred embodiment of the present invention, a learning unit is further included. The learning unit reads sample image data and screen frequency range, sets the display screen at different refresh frequency points in a certain step manner, and displays standard images of 5 colors in sequence. Each display image and frequency combination is used as a set of display conditions.

[0012] In a preferred embodiment of the present invention, a photoelectric sensor is further included. The learning unit reads the brightness change waveform signal collected by the photoelectric sensor and adjusts the sampling parameters of the visual sensor based on the extraction and calculation of image features.

[0013] In a preferred embodiment of the present invention, the algorithm model employs a multi-layer neural network, including an input layer, a convolutional layer, a first hidden layer, a second hidden layer, a fully connected layer, and an output layer.

[0014] In a preferred embodiment of the present invention, the input layer comprises three parts: the visual sensor sampling rate Si at time i; the visual sensor parameter features composed of the sampling duty cycle Pi; and the data of the visual sensor sampled image at time i in the red channel, green channel, and blue channel.

[0015] In a preferred embodiment of the present invention, the output layer includes two neurons, which are the sampling rate and duty cycle of the visual sensor predicted by the model for the next time step.

[0016] In a preferred embodiment of the present invention, the algorithm model is trained using the forward and backward propagation methods of a neural network. Training is stopped when the loss function Los s for all image samples and the entire frequency range of the prediction in the system is less than a specified value δ, and δ is not greater than the error in the display test standard.

[0017] The technical solution of the present invention has the following advantages over the prior art:

[0018] (1) This application supports adaptive learning for different display screen types, different visual sensor types and different frequency ranges, and learns automatically as the scene changes, with high robustness.

[0019] (2) This application has a low dependence on measuring instruments. The measurement of the display screen properties only relies on the vision sensor, which is time-saving and efficient.

[0020] (3) This application can achieve automated measurement without the need for the display to provide a software control interface for the refresh rate, and has a low dependence on the display software. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, some of the drawings in the following description are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of texture noise caused by sampling misalignment in an embodiment of the present invention;

[0023] Figure 2 This is the overall process for measuring dynamic solid color images on a screen using pure visual perception in this embodiment of the invention;

[0024] Figure 3 This is a schematic diagram of the adaptive learning system for visual sensor sampling parameters in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the adaptive learning algorithm model for sampling parameters in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of time synchronization alignment search under each step of the prediction parameters in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of a purely visual screen attribute measurement system in an embodiment of the present invention. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0029] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0030] Example 1

[0031] See Figure 1-6As shown, this invention proposes an adaptive learning system for visual sensor sampling parameters. This system can adaptively adjust the sampling rate and sampling duration of the visual sensor through algorithms and software. Even for display screens with unknown refresh rates, it can automatically search for frequencies to achieve frequency synchronization and thus accurately perform visual sampling. It can be used in applications such as automated measurement of displays and portable measurement of VR devices. The system includes a set of standard sample images, a standard display screen, a photoelectric signal sensor, a visual sensor, display screen refresh rate range setting software, and a learning algorithm software. The standard sample images are divided into five standard colors: red, green, blue, black, and white. The display screen refresh rate setting software is used to set the minimum and maximum refresh rates of the screen. The learning algorithm software reads the sample image data and the screen frequency range, and then sets the display screen to different refresh rate points in a certain stepwise manner, sequentially displaying the five standard images of different colors. Each display image and frequency combination serves as a set of display conditions. Under each set of display conditions, the learning unit has built-in learning algorithm software, which simultaneously reads the image data collected by the visual sensor as the image sampling result of the display. Simultaneously, the learning algorithm software also reads the brightness change waveform signal collected by the photoelectric sensor. Based on the extraction and calculation of image features, the learning algorithm software adjusts the sampling parameters of the visual sensor. When the waveform error between the visual sensor sampling parameters output by the learning algorithm software and the photoelectric sensor is less than a certain value under all combined conditions, the learning task is deemed to have met the standard, and the model is finalized.

[0032] The learning algorithm software employs deep learning-based algorithm models, such as... Figure 4 As shown, the model uses a multi-layer neural network, including an input layer ( Figure 4 14) Convolutional layer ( Figure 4 15 marked in the middle), hidden layer 1 ( Figure 4 16) Hidden layer 2 ( Figure 4 17) Fully connected layer ( Figure 4 (18) and output layer ( Figure 4 (19 marked in the middle).

[0033] The input layer is divided into four parts, namely the visual sensor sampling rate Si at time i. Figure 4 The 20 marked in the middle), sampling duty cycle Pi ( Figure 4 The visual sensor parameter characteristics constituted by the 21 marked in the figure, the visual sensor sampled image at time i in the red channel data ( Figure 4 (22 marked in the middle), green channel data ( Figure 4 (23) and blue channel data ( Figure 4 As indicated in point 24), Si is normalized using Sia / Smax for its value.

[0034] Si = Si a / Smax

[0035] Here, Si a is the absolute value of the current sampling rate, Smax is the maximum sampling rate supported by the system, and Pi is the ratio of the sampling duration in each frame to the total duration of a frame, with a value range of [0,1]. The image data of the three color channels are normalized using the pixel value Vi / 255 of each pixel.

[0036] Vi = Vi a / 255

[0037] Vi a represents the grayscale value of each pixel in each channel, with a maximum value of 255 for each pixel. The four parts of the input layer data constitute an input vector, represented as [Si,Pi],Red[M*N],Green[M*N],Blue[M*N]]. M and N in the vector are the number of pixels in the row and column of the image sampled by the vision sensor. Correspondingly, the number of neurons in the input layer is 2+3*M*N.

[0038] The convolutional layer is divided into two parts: a nonlinear mapping part for the sampling parameters of the visual sensor and a convolution operation part for the image data. The neurons in the former are connected to the outputs of the sampling parameter neurons in the input layer, and the neurons in the latter are connected to the outputs of the three channel neurons in the input layer. Convolution operations are performed on the image to obtain features such as edges.

[0039] Hidden layers 1 and 2 contain twice the number of neurons as the previous layer and employ a recurrent structure, meaning that the output of each neuron simultaneously serves as the input for the next time step. This serializes the image's 3-channel data, effectively capturing the variation patterns of image texture noise over the past L time steps, including four patterns: top-to-bottom variation, bottom-to-top variation, stationary, and intermittent.

[0040] The number of neurons in the fully connected layer is 1.5 times that of the hidden layer 2. Each neuron in the fully connected layer is connected to the neurons in the previous layer, which plays a role in non-linear mapping.

[0041] The output layer consists of two neurons: the sampling rate Spredict and the duty cycle Ppredict of the visual sensor at the next time step, which are predicted by the model.

[0042] When learning algorithms, the process is divided into two stages: model training and inference.

[0043] The model is trained by an adaptive learning system that uses visual sensor sampling parameters. During training, the refresh rate fi of the display is randomly set, and the input sample image f[t]i is the display's brightness change waveform signal Wi, which is sensed by a photoelectric sensor. Figure 5(27) The model's loss function is defined as Loss.

[0044] Loss=argmin[(Spredict-Swi)2+(Ppredict-Pwi)2]

[0045] Where Swi = fi, is the measured display refresh rate, Pwi is the proportion of the high-level time in the waveform signal Wi within one period 1 / Swi, and the loss function is the waveform predicted by the system ( Figure 5 The sum of squared errors of the two parameters marked in the figure (28) and the two measured parameters;

[0046] In the formula, argmin represents the output parameters Spredict and Ppredict for each step of the system. Keeping these values ​​constant, the visual sensor searches within a certain time range using actions, performing time-synchronized search and comparison. Figure 5 (29) Time synchronization search and comparison refers to adjusting the starting point of the visual sensor's sampling period to determine if it can be aligned with the starting point of Wi's period. During the search process, the visual sensor's sampling action will increment by an offset of β within the [t+1 / Spredict] time window, such as... Figure 5 As shown.

[0047] During training, the forward and backward propagation methods of the neural network are used. The training stops when the loss function Loss of the system for all image samples and the entire frequency range is less than a specified value δ, and δ is not greater than the error definition in the display test standard.

[0048] In the inference phase, the system composition is as follows: Figure 6 As shown, the adaptive learning algorithm software specifies the display content as a certain type of solid color image in real time without setting the display refresh rate. It sends the current sampling parameters of the vision sensor and the image into the model, automatically obtains the ideal setting parameters of the vision sensor and modifies them into the system prediction values, thereby obtaining an accurate sample of the current screen image for subsequent calculation and measurement of attributes such as brightness and chromaticity.

[0049] According to an embodiment of the present invention, standard sample image data is acquired through a visual sensor, and adaptive learning is performed to obtain an algorithm model for this type of visual sensor; specifically including:

[0050] Set the minimum and maximum refresh rates for the display screen;

[0051] Read the sample image and display screen frequency range, set the screen to different refresh rate points in a certain step and display a standard image of 5 colors;

[0052] Image data is acquired via a visual sensor under each display condition;

[0053] Brightness change waveform signals were acquired using a photoelectric sensor;

[0054] The visual sensor extracts and calculates features from the images it acquires, and adjusts the visual sensor's sampling parameters.

[0055] The learning process stops when the frequency output by the algorithm matches the waveform of the photoelectric sensor.

[0056]

[0057] Table 1 Input Data Types

[0058]

[0059] Table 2 Output Data Types

[0060] In summary, this application supports adaptive learning for different display screen types, different vision sensor types, and different frequency ranges. It learns automatically as the scene changes, exhibits high robustness, has low dependence on measuring instruments, and the measurement of display screen attributes relies solely on the vision sensor, resulting in short measurement time and high efficiency. Automated measurement can be achieved without the display providing a software control interface for refresh rate, thus having low dependence on display software.

[0061] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0062] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to the above embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0063] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual sensor sampling parameter adaptive learning system, characterized in that, include: Visual sensors; Acquire standard sample images for optical testing, perform adaptive learning of visual sensor sampling parameters, and obtain an algorithm model for the corresponding category of visual sensors; Image features are obtained through an algorithm model, and the sampling parameters of the visual sensor are automatically adjusted to achieve synchronous sampling with the screen refresh action; The algorithm model employs a multi-layer neural network; The standard sample image includes five standard colors: red, green, blue, black, and white. It also includes a learning unit, which reads sample image data and screen frequency range, sets the display screen at different refresh rate points in a stepwise manner, and displays standard images of 5 colors in sequence. Each display image and frequency combination is used as a set of display conditions. It also includes a photoelectric sensor. The learning unit reads the brightness change waveform signal collected by the photoelectric sensor and adjusts the sampling parameters of the vision sensor based on the extraction and calculation of image features. The algorithm model is trained using the forward and backward propagation methods of neural networks. Training stops when the loss function Loss for all image samples and the entire frequency range of the system is less than the specified value δ, and δ is not greater than the error in the display test standard.

2. The visual sensor sampling parameter adaptive learning system according to claim 1, characterized in that, The algorithm model employs a multi-layer neural network, including an input layer, a convolutional layer, a first hidden layer, a second hidden layer, a fully connected layer, and an output layer.

3. The visual sensor sampling parameter adaptive learning system according to claim 2, characterized in that, The input layer consists of three parts: the visual sensor sampling rate Si at time i; and the visual sensor parameter characteristics composed of the sampling duty cycle Pi. The visual sensor samples the image at time i using data in the red channel, green channel, and blue channel.

4. The visual sensor sampling parameter adaptive learning system according to claim 2, characterized in that, The output layer consists of two neurons, which represent the sampling rate and duty cycle of the visual sensor at the next time step predicted by the model.

Images