Image processing system with image sensor

Abstract

The present invention relates to the field of image processing technology, and in particular to an image processing system with an image sensor, including an image sensor and preprocessing module, a time-coded imaging module, a feature extraction and recognition module, an image processing and analysis module, and a display and storage module, wherein: the image sensor and preprocessing module capture image data; the time-coded imaging module realizes the capture and analysis of fast-moving targets; the feature extraction and recognition module extracts image features and performs image recognition; the image processing and analysis module processes and analyzes the results of image recognition, including object tracking, behavior analysis, and scene understanding; the display and storage module is used to display the analysis results, and store the analysis results in a storage device for subsequent call and analysis. The present invention greatly improves the accuracy and reliability of the system in capturing high-speed moving targets.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing system with an image sensor. Background Art

[0002] With the rapid development of computer vision technology, image processing systems have been widely used in various fields, such as intelligent monitoring, autonomous driving, industrial inspection, etc. These systems rely on high-quality image data capture, precise motion target analysis, and detailed result display and storage to achieve the understanding and processing of complex scenes. However, in dynamic and complex environments, especially for the capture and analysis of high-speed moving targets, traditional image processing systems still face many challenges.

[0003] Traditional image processing systems often have difficulty ensuring image clarity and accuracy when processing high-speed moving targets. Under a variety of lighting conditions, the images captured by image sensors are of low quality and are often accompanied by noise and color distortion, which affects the subsequent image processing effects. Due to the lack of precise time information and high frame rate imaging technology, the system is prone to motion artifacts when detecting high-speed moving targets, and it is difficult to distinguish between moving objects and stationary backgrounds. Existing systems often lack efficient algorithm support when performing three-dimensional motion trajectory reconstruction and scene understanding, resulting in low processing efficiency and difficulty in meeting the needs of real-time applications.

[0004] The present invention provides an image processing system with an image sensor to overcome the deficiencies of the prior art, especially in complex and dynamic environments, to improve the efficiency of capturing and processing image data, and to enable the system to exhibit excellent performance and wide applicability in complex application scenarios. Summary of the invention

[0005] Based on the above objectives, the present invention provides an image processing system having an image sensor.

[0006] An image processing system with an image sensor comprises an image sensor and preprocessing module, a time coding imaging module, a feature extraction and recognition module, an image processing and analysis module, and a display and storage module, wherein;

[0007] The image sensor and the preprocessing module capture image data and preprocess the received image data, the preprocessing including denoising, white balance and color correction;

[0008] The time-coded imaging module encodes time information into the preprocessed image data to achieve the capture and analysis of fast-moving targets, specifically including:

[0009] Time coding: During the image data capture process, the time information is encoded into each frame of image data, so that the image data contains timestamp information;

[0010] Motion detection: connected with time coding, it detects moving targets in image data based on time coding information, distinguishes moving objects from static background, and reduces motion artifacts;

[0011] High frame rate imaging: Achieve high frame rate image capture and support image sequences with high temporal resolution;

[0012] Spatiotemporal reconstruction: Spatiotemporal reconstruction based on time coding information and high frame rate image data, providing three-dimensional motion trajectory reconstruction;

[0013] The feature extraction and recognition module extracts image features, including edge features, texture features, and shape features, from the image data after the temporal information is encoded, and performs image recognition based on the extracted image features, including classification, target detection, and semantic segmentation;

[0014] The image processing and analysis module processes and analyzes the results of image recognition, including object tracking, behavior analysis and scene understanding;

[0015] The display and storage module is used to display the analysis results and store the analysis results in a storage device for subsequent calling and analysis.

[0016] Furthermore, the image sensor and preprocessing module include:

[0017] Image capture: Use image sensor to capture image data, the image sensor is CMOS sensor;

[0018] Image preprocessing: Preprocess the received image data, including:

[0019] Denoising: Use an adaptive filtering algorithm to denoise the image data to reduce random noise in the image data. The calculation formula is:

[0020]

[0021] Among them, I filtered (x, y) is the denoised image value, I(x+i, y+j) is the pixel value of the original image, w(i, j) is the weight of the filter, and k is the size of the filter window;

[0022] White balance: adjusts the color balance in the image through the gray world algorithm to correct the influence of the light source color temperature. The calculation formula is:

[0023]

[0024] Among them, R(x, y), G(x, y) and B(x, y) are the values ​​of the red, green and blue channels of the image at position (x, y), respectively.avg , G avg and B avg is the average value of the red, green, and blue channels in the image, and N is the total number of pixels in the image;

[0025] Adjusted white balance value:

[0026]

[0027] Where R′(x, y) and B′(x, y) are the adjusted values ​​of the red and blue channels;

[0028] Color correction: Use a color correction algorithm based on a lookup table to perform color correction on the image to correct the color deviation caused by changes in sensor characteristics or lighting conditions. The calculation formula is:

[0029] R′=LUT R (R);

[0030] G′=LUT G (G);

[0031] B′=LUT B (B);

[0032] Among them, R, G, and B are the values ​​of the red, green, and blue channels of the original image, respectively, and R', G', and B' are the values ​​of the red, green, and blue channels after color correction, respectively. R , LUT G and LUTs B These are the lookup tables for the red, green, and blue channels respectively.

[0033] Furthermore, the time coding includes:

[0034] Timestamp generation: When the image sensor captures each frame of image data, the image processing system clock generates the corresponding timestamp information T i , where i represents the frame number, and the timestamp records the time when the corresponding frame image data is captured;

[0035] Timestamp embedding: The timestamp information is encoded into the image frame through the Alpha channel. The calculation formula is:

[0036] I′(x,y,α)={I(x,y),T i};

[0037] Among them, α represents the additional channel;

[0038] Time synchronization: Ensure the synchronization of timestamp information and image data, and keep the clock of the image processing system consistent with the capture frequency of the image sensor to avoid offset and error of time information.

[0039] Further, the motion detection includes:

[0040] Time difference calculation: Use the timestamp information provided by the time code to perform time difference calculation on the image data of adjacent frames to obtain motion information. The calculation formula is:

[0041] D t (x, y)=I(x, y, t)-I(x, y, t-Δt);

[0042] Among them, I(x, y, t) and I(x, y, t-Δt) are the image frames with timestamps t and t-Δt, respectively. t (x, y) is the time difference result of the corresponding pixel position;

[0043] Threshold segmentation: Perform threshold segmentation on the time difference result to distinguish the moving object from the static background. The calculation formula is:

[0044]

[0045] Wherein, τ is the preset threshold, M(x, y) is the binarized motion detection result, 1 indicates motion is detected, and 0 indicates stillness;

[0046] Artifact removal: By using time coding information and analyzing the motion consistency between consecutive frames, artifacts caused by noise or illumination changes are eliminated. The calculation formula is:

[0047] M'(x, y)=M(x, y)∩M(x, y, t+Δt);

[0048] Wherein, M′(x, y) is the motion detection result after eliminating artifacts.

[0049] Furthermore, the high frame rate imaging includes:

[0050] High-speed image sensor: Use high-speed image sensor to capture high frame rate image data;

[0051] Fast data readout: Transmit captured high frame rate image data via low voltage differential signaling (LVDS);

[0052] Cache storage: Temporarily store captured high frame rate image data to avoid delays and losses during data transmission;

[0053] Frame synchronization control: Through the clock synchronization mechanism, the time consistency of each frame of image data during capture and transmission is ensured to avoid time deviation between frames. The calculation formula is:

[0054] t i+1 =t i +Δt;

[0055] Among them, t i and t i+1 They represent the timestamps of the i-th frame and the i+1-th frame respectively, and Δt is the time interval between frames.

[0056] Furthermore, the spatiotemporal reconstruction includes:

[0057] Time series analysis: Use the timestamp information provided by the time code to perform time series analysis on the high frame rate image data to determine the image data sequence at each time point. The calculation formula is:

[0058] S(t)={I(x,y,t)|t=t0,t1,...,t n};

[0059] Where S(t) represents the image sequence corresponding to time t, and I(x, y, t) is the image data at time t;

[0060] Motion vector calculation: Calculate the motion vector (u, v) of each pixel based on the optical flow algorithm to obtain the motion information of the image. The calculation formula is:

[0061] I x u+I y v+I t =0;

[0062] Among them, I x is the partial derivative of image brightness in the x direction, I y is the partial derivative of image brightness in the y direction, I t is the partial derivative of the image brightness at time t, u is the velocity component of the optical flow in the x direction, and v is the velocity component of the optical flow in the y direction;

[0063] Three-dimensional coordinate reconstruction: Using multi-viewpoint stereo vision method, three-dimensional coordinate reconstruction is performed through high-frame rate image data from multiple perspectives. The calculation formula is:

[0064]

[0065] Where n is the number of viewpoints, f i and B i are the focal length and baseline distance of the i-th viewpoint, respectively. and is the pixel coordinate in the i-th viewpoint;

[0066] Trajectory reconstruction: Combine the particle filter algorithm to smooth and predict the reconstructed three-dimensional trajectory.

[0067] Furthermore, the particle filtering algorithm includes:

[0068] Initialization: Generate N particles and give the same initial weights The calculation formula for position initialization is:

[0069]

[0070] Among them, x0, y0, z0 are the initial positions, is the normally distributed noise, σ x , σ y , σ z is the standard deviation of the initial position noise;

[0071] Particle prediction: predict particles based on the state transition model, combine time coding information and motion vector calculation results to update the state, including position prediction and motion vector update;

[0072] The calculation formula for the position prediction is:

[0073]

[0074] in, is the predicted position of particle i at time k, is the predicted position of particle i at time k-1, is the motion vector component of the particle, Δt is the time interval;

[0075] The calculation formula for the motion vector update is:

[0076]

[0077]

[0078] Update weights: Update the weight of each particle according to the observation model, and adjust the weights based on the matching degree of time coding information and image data. The calculation formula is:

[0079]

[0080] in, is the probability of observing particle i at time k, z k is the actual observed value at time k, is the predicted observation value of particle i, σ z is the standard deviation of the observation noise;

[0081]

[0082] in, is the weight of particle i at time k, is the weight of particle i at time k-1;

[0083] Normalized weight: Normalize the weight, the calculation formula is:

[0084]

[0085] in, is the normalized weight of particle i at time k;

[0086] Resampling: Calculate the cumulative weight C based on the normalized weight j , generate a uniformly distributed random number u, and for each particle i, find j such that C j-1 <u<C j , select particle j as the new particle, and the calculation formula is:

[0087]

[0088] in, is a random number uniformly distributed in the interval [0, 1];

[0089] State estimation: The state estimation of the image processing system is calculated based on the resampled particles, including position estimation and motion vector estimation;

[0090] The calculation formula for the position estimation is:

[0091]

[0092]

[0093] in, is the position estimate at time k;

[0094] The calculation formula for the motion vector estimation is:

[0095]

[0096] in, is the motion vector estimate at time k.

[0097] Furthermore, the feature extraction and recognition module includes:

[0098] Edge feature extraction: Use the Sobel operator to extract edge features from image data. The calculation formula is:

[0099]

[0100] Among them, G x and G y are the gradients of the image in the x and y directions respectively, and G is the edge strength;

[0101] Texture feature extraction: Use the gray level co-occurrence matrix (GLCM) to extract the texture features of image data. The calculation formula is:

[0102]

[0103] Where P(i, j) is the probability of gray levels i and j appearing together in the gray level co-occurrence matrix;

[0104] Shape feature extraction: The Fourier Descriptors method is used to extract the shape features of image data. The calculation formula is:

[0105]

[0106] Among them, F(u) is the Fourier descriptor, f(n) is the point on the image contour, N is the total number of contour points, j is the imaginary unit, and u is the frequency component;

[0107] Image recognition: Use convolutional neural network (CNN) models to perform image data classification, object detection, and semantic segmentation based on the extracted image features.

[0108] Furthermore, the convolutional neural network (CNN) model includes:

[0109] Convolution layer: Perform convolution operation on the input image data to extract low-level features (such as edges and textures). The calculation formula is:

[0110] (I*K)(x,y)=∑ i ∑ j I(xi,yj)K(i,j);

[0111] Where I is the input image, K is the convolution kernel, and (x, y) is the location of the output feature map;

[0112] Pooling layer: Perform the maximum pooling operation on the convolutional feature map to reduce the size of the feature map. The calculation formula is:

[0113] P(x, y) = max i，j I(x+i,y+j);

[0114] Among them, P(x, y) is the feature map after pooling, and the max operation is to take the maximum value in the pooling window;

[0115] Fully connected layer: Expand the pooled feature map into a one-dimensional vector, perform a fully connected operation, and output the classification result. The calculation formula is:

[0116]

[0117] in, is the classification result, W is the weight matrix, x is the input feature vector, and b is the bias;

[0118] Loss function: The convolutional neural network model is trained using the cross entropy loss function, and the calculation formula is:

[0119]

[0120] Among them, L is the loss value, y i is the true label, is the predicted probability.

[0121] Furthermore, the image processing and analysis module includes:

[0122] Object tracking: Based on the results of image recognition, the position and bounding box of the initial target are determined, and the Kalman filter algorithm is used to track the target;

[0123] Behavior analysis: Extract the target's motion trajectory and behavior characteristics (such as speed, acceleration, direction change, etc.) from the object tracking results, classify the extracted behavior characteristics, and identify the target's behavior pattern (such as walking, running, staying, etc.);

[0124] Scene segmentation: Perform scene segmentation on image data and label each part of the image;

[0125] Scene analysis: Based on the results of scene segmentation, scene analysis is performed to extract elements in the scene (such as people, cars, buildings, etc.) and understand the semantic information of the scene.

[0126] Beneficial effects of the present invention:

[0127] The present invention, through the image sensor and the preprocessing module, the system can capture high-quality image data under a variety of lighting conditions, and perform denoising, white balance and color correction to ensure that the image data has high quality and high fidelity, which provides a reliable data basis for subsequent time coding and image processing, and significantly improves the system's image capture and processing capabilities in complex environments.

[0128] The present invention embeds precise time information into image data to achieve the capture and analysis of fast-moving targets. By combining high frame rate imaging and spatiotemporal reconstruction technology, the system can accurately detect moving targets in images, reduce motion artifacts, and reconstruct three-dimensional motion trajectories through a particle filtering algorithm to provide high-precision motion analysis, which greatly improves the accuracy and reliability of the system in capturing high-speed moving targets.

[0129] The present invention extracts rich image features from image data encoded with time information, and performs image recognition based on these features, including classification, target detection and semantic segmentation. The image processing and analysis module further processes and analyzes the recognition results to achieve object tracking, behavior analysis and scene understanding. Finally, the display and storage module displays the analysis results intuitively and stores them in a storage device for subsequent call and analysis. The entire system provides detailed analysis results while efficiently processing image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0130] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0131] Figure 1 A schematic diagram of system function modules according to an embodiment of the present invention;

[0132] Figure 2 Schematic diagram of a time-encoded imaging module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0133] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0134] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0135] like Figure 1-Figure 2 As shown, an image processing system with an image sensor includes an image sensor and preprocessing module, a time coding imaging module, a feature extraction and recognition module, an image processing and analysis module, and a display and storage module, wherein;

[0136] The image sensor and the preprocessing module capture image data and preprocess the received image data, and the preprocessing includes denoising, white balance and color correction;

[0137] The time-coded imaging module encodes time information into preprocessed image data to capture and analyze fast-moving targets. Specifically, it includes:

[0138] Time encoding: During the image data capture process, the time information is encoded into each frame of image data so that the image data contains timestamp information for subsequent processing;

[0139] Motion detection: connected with time coding, it detects moving targets in image data based on time coding information, distinguishes moving objects from static background, and reduces motion artifacts;

[0140] High frame rate imaging: Achieve high frame rate image capture, support high temporal resolution image sequences, and ensure clear capture of high-speed moving targets;

[0141] Spatiotemporal reconstruction: Spatiotemporal reconstruction based on time-coded information and high-frame-rate image data, providing three-dimensional motion trajectory reconstruction, suitable for motion analysis and industrial detection;

[0142] The feature extraction and recognition module extracts image features, including edge features, texture features, and shape features, from the image data after temporal information encoding, and performs image recognition based on the extracted image features, including classification, target detection, and semantic segmentation;

[0143] The image processing and analysis module processes and analyzes the results of image recognition, including object tracking, behavior analysis, and scene understanding;

[0144] The display and storage module is used to display the analysis results and store the analysis results in a storage device for subsequent call and analysis;

[0145] Through the above content, efficient image data capture and processing, accurate moving target analysis and identification, and detailed result display and storage are achieved. The introduction of the time-coded imaging module significantly improves the accuracy and reliability of the system in high-speed moving target capture, reduces motion artifacts, and enhances the accuracy of three-dimensional motion trajectory reconstruction, enabling the system to demonstrate excellent performance and wide applicability in complex application scenarios.

[0146] Image sensor and pre-processing module include:

[0147] Image capture: Image data is captured using an image sensor, which is a CMOS sensor capable of capturing high-quality image data under a variety of lighting conditions;

[0148] Image preprocessing: Preprocess the received image data, including:

[0149] Denoising: Use an adaptive filtering algorithm to denoise the image data to reduce random noise in the image data and improve the signal-to-noise ratio of the image. The calculation formula is:

[0150]

[0151] Among them, I filtered (x, y) is the denoised image value, I(x+i, y+j) is the pixel value of the original image, w(i, j) is the weight of the filter, the weight is adaptively adjusted according to the noise level, and k is the size of the filter window;

[0152] White balance: adjust the color balance in the image through the gray world algorithm, correct the influence of the light source color temperature, and make the color in the image more natural and accurate. The calculation formula is:

[0153]

[0154] Among them, R(x, y), G(x, y) and B(x, y) are the values ​​of the red, green and blue channels of the image at position (x, y), respectively. avg , G avg and B avg is the average value of the red, green, and blue channels in the image, and N is the total number of pixels in the image;

[0155] Adjusted white balance value:

[0156]

[0157] Where R′(x, y) and B′(x, y) are the adjusted values ​​of the red and blue channels;

[0158] Color correction: Use a color correction algorithm based on a lookup table to perform color correction on the image to correct the color deviation caused by changes in sensor characteristics or lighting conditions, making the image color more realistic and consistent. The calculation formula is:

[0159] R′=LUT R (R);

[0160] G′=LUT G (G);

[0161] B′=LUT B (B);

[0162] Among them, R, G, and B are the values ​​of the red, green, and blue channels of the original image, respectively, and R', G', and B' are the values ​​of the red, green, and blue channels after color correction, respectively.R , LUT G and LUTs B These are the lookup tables for the red, green, and blue channels, respectively, used to perform color correction on each channel;

[0163] Through the above steps, the captured image data is ensured to be of high quality and high fidelity, providing a reliable data basis for subsequent time coding and image processing.

[0164] Time codes include:

[0165] Timestamp generation: When the image sensor captures each frame of image data, the image processing system clock generates the corresponding timestamp information T i , where i represents the frame number, and the timestamp records the time when the corresponding frame image data is captured;

[0166] Timestamp embedding: The timestamp information is encoded into the image frame through the Alpha channel. The calculation formula is:

[0167] I′(x,y,α)={I(x,y),T i};

[0168] Among them, α represents an additional channel, which is used to store timestamp information;

[0169] Time synchronization: ensure the synchronization of timestamp information and image data, and keep the clock of the image processing system consistent with the capture frequency of the image sensor to avoid offset and error of time information;

[0170] Through the above steps, the time encoding unit can encode accurate time information into each frame of image data, so that the image data contains timestamp information, providing accurate time reference for subsequent motion detection, high frame rate imaging and spatiotemporal reconstruction.

[0171] Motion detection includes:

[0172] Time difference calculation: Use the timestamp information provided by the time code to perform time difference calculation on the image data of adjacent frames to obtain motion information. The calculation formula is:

[0173] D t (x, y)=I(x, y, t)-I(x, y, t-Δt);

[0174] Among them, I(x, y, t) and I(x, y, t-Δt) are the image frames with timestamps t and t-Δt, respectively. t (x, y) is the time difference result of the corresponding pixel position;

[0175] Threshold segmentation: Perform threshold segmentation on the time difference result to distinguish the moving object from the static background. The calculation formula is:

[0176]

[0177] Wherein, τ is the preset threshold, M(x, y) is the binarized motion detection result, 1 indicates motion is detected, and 0 indicates stillness;

[0178] Artifact removal: By using time coding information and analyzing the motion consistency between consecutive frames, artifacts caused by noise or illumination changes are eliminated. The calculation formula is:

[0179] M'(x, y)=M(x, y)∩M(x, y, t+Δt);

[0180] Where M′(x, y) is the motion detection result after removing artifacts, indicating consistent motion detection between consecutive frames;

[0181] Through the above steps, motion detection can accurately detect moving targets in image data based on time coding information, effectively distinguish moving objects from static backgrounds, reduce motion artifacts, and improve the accuracy and reliability of motion detection.

[0182] High frame rate imaging includes:

[0183] High-speed image sensor: Use high-speed image sensors to capture high-frame rate image data in a short time. The frame rate of high-speed image sensors can reach hundreds to thousands of frames per second to ensure high temporal resolution.

[0184] Fast data readout: High frame rate image data captured in a short time is transmitted through low voltage differential signal (LVDS) to ensure real-time data transmission;

[0185] Cache storage: Temporarily stores captured high frame rate image data to avoid delays and losses during data transmission. The cache capacity and speed must match the sensor frame rate to support continuous high frame rate image capture.

[0186] Frame synchronization control: Through the clock synchronization mechanism, the time consistency of each frame of image data during capture and transmission is ensured to avoid time deviation between frames. The calculation formula is:

[0187] t i+1 =t i +Δt;

[0188] Among them, t i and t i+1 They represent the timestamps of the i-th frame and the i+1-th frame respectively, and Δt is the time interval between frames, which is controlled by the system clock;

[0189] Through the above steps, high frame rate imaging can achieve high frame rate image capture, support high temporal resolution image sequences, and provide a high-quality image data foundation for subsequent motion analysis and spatiotemporal reconstruction.

[0190] Space-time reconstruction includes:

[0191] Time series analysis: Use the timestamp information provided by the time code to perform time series analysis on the high frame rate image data to determine the image data sequence at each time point. The calculation formula is:

[0192] S(t)={I(x,y,t)|t=t0,t1,...,t n};

[0193] Where S(t) represents the image sequence corresponding to time t, and I(x, y, t) is the image data at time t;

[0194] Motion vector calculation: Calculate the motion vector (u, v) of each pixel based on the optical flow algorithm to obtain the motion information of the image. The calculation formula is:

[0195] I x u+I y v+I t =0;

[0196] Among them, I x is the partial derivative of the image brightness in the x direction, indicating the rate at which the image brightness changes in the x direction, I y is the partial derivative of the image brightness in the y direction, indicating the rate at which the image brightness changes in the y direction, I t is the partial derivative of the image brightness at time t, indicating the rate at which the image brightness changes over time, u is the velocity component of the optical flow in the x direction, indicating the speed of the pixel moving in the x direction, and v is the velocity component of the optical flow in the y direction, indicating the speed of the pixel moving in the y direction;

[0197] Three-dimensional coordinate reconstruction: Using multi-viewpoint stereo vision method, three-dimensional coordinate reconstruction is performed through high-frame rate image data from multiple perspectives. The calculation formula is:

[0198]

[0199] Where n is the number of viewpoints, f i and B i are the focal length and baseline distance of the i-th viewpoint, respectively. and is the pixel coordinate in the i-th viewpoint;

[0200] Trajectory reconstruction: Combine the particle filter algorithm to smooth and predict the reconstructed three-dimensional trajectory to improve the stability and robustness of the trajectory;

[0201] Through the above improvements, spatiotemporal reconstruction can be performed more accurately based on time coding information and high frame rate image data, providing high-precision three-dimensional motion trajectory reconstruction, which is suitable for application scenarios such as motion analysis and industrial inspection.

[0202] The particle filter algorithm includes:

[0203] Initialization: Generate N particles and give the same initial weights The calculation formula for position initialization is:

[0204]

[0205] Among them, x0, y0, z0 are the initial positions, is the normally distributed noise, σ x , σ y , σ z is the standard deviation of the initial position noise;

[0206] Particle prediction: predict particles based on the state transition model, combine time coding information and motion vector calculation results to update the state, including position prediction and motion vector update;

[0207] The calculation formula for position prediction is:

[0208]

[0209] in, is the predicted position of particle i at time k, is the predicted position of particle i at time k-1, is the motion vector component of the particle, Δt is the time interval, and normal distribution noise is used to simulate process noise;

[0210] The calculation formula for motion vector update is:

[0211]

[0212] Update weights: Update the weight of each particle according to the observation model, and adjust the weights based on the matching degree of time coding information and image data. The calculation formula is:

[0213] Observation model:

[0214] in, is the probability of observing particle i at time k, z k is the actual observed value at time k, is the predicted observation value of particle i, σ z is the standard deviation of the observation noise;

[0215] Weight adjustment:

[0216] in, is the weight of particle i at time k, is the weight of particle i at time k-1;

[0217] Normalized weights: Normalize the weights to ensure that the sum of the weights is 1. The calculation formula is:

[0218]

[0219] in, is the normalized weight of particle i at time k;

[0220] Resampling: Calculate the cumulative weight C based on the normalized weight j , generate a uniformly distributed random number u, and for each particle i, find j such that C j-1 <u<C j , select particle j as the new particle to eliminate particles with smaller weights and enhance particles with larger weights. The calculation formula is:

[0221]

[0222] in, is a random number uniformly distributed in the interval [0, 1];

[0223] State estimation: The state estimation of the image processing system is calculated based on the resampled particles, including position estimation and motion vector estimation;

[0224] The position estimate is calculated as:

[0225]

[0226] in, is the position estimate at time k;

[0227] The calculation formula for motion vector estimation is:

[0228]

[0229] in, is the motion vector estimate at time k;

[0230] Through the above, the particle filter algorithm can smooth and predict the three-dimensional motion trajectory more accurately, and combine high frame rate image data and time coding information to achieve three-dimensional trajectory reconstruction with higher accuracy and robustness.

[0231] The feature extraction and recognition module includes:

[0232] Edge feature extraction: Use the Sobel operator to extract edge features from image data. The calculation formula is:

[0233]

[0234] Among them, Gx and Gy are the gradients of the image in the x and y directions respectively, and G is the edge strength;

[0235] Texture feature extraction: Use the gray level co-occurrence matrix (GLCM) to extract the texture features of image data. The calculation formula is:

[0236]

[0237] Where P(i, j) is the probability of gray levels i and j appearing together in the gray level co-occurrence matrix;

[0238] Shape feature extraction: The Fourier Descriptors method is used to extract the shape features of image data. The calculation formula is:

[0239]

[0240] Among them, F(u) is the Fourier descriptor, f(n) is the point on the image contour, N is the total number of contour points, j is the imaginary unit, and u is the frequency component;

[0241] Image recognition: Use convolutional neural network (CNN) models to perform image data classification, object detection, and semantic segmentation based on extracted image features;

[0242] Through the above steps, rich image features can be extracted from the image data after the time information is encoded, and accurate image recognition can be performed based on these features.

[0243] The Convolutional Neural Network (CNN) model includes:

[0244] Convolution layer: Perform convolution operation on the input image data to extract low-level features (such as edges and textures). The calculation formula is:

[0245] (I*K)(x,y)=∑ i ∑ j I(xi,yj)K(i,j);

[0246] Where I is the input image, K is the convolution kernel, and (x, y) is the location of the output feature map;

[0247] Pooling layer: Perform the maximum pooling operation on the convolutional feature map to reduce the size of the feature map. The calculation formula is:

[0248] P(x, y) = max i，j I(x+i,y+j);

[0249] Among them, P(x, y) is the feature map after pooling, and the max operation is to take the maximum value in the pooling window;

[0250] Fully connected layer: Expand the pooled feature map into a one-dimensional vector, perform a fully connected operation, and output the classification result. The calculation formula is:

[0251]

[0252] in, is the classification result, W is the weight matrix, x is the input feature vector, and b is the bias;

[0253] The calculation formula of the softmax function is:

[0254]

[0255] in, is the class probability of pixel (i, j), z ij is the output of the convolutional neural network model, representing the category score at position (i, j), and K is the number of categories;

[0256] Loss function: The convolutional neural network model is trained using the cross entropy loss function, and the calculation formula is:

[0257]

[0258] Among them, L is the loss value, y i is the true label, is the predicted probability;

[0259] Through the above steps, rich image features can be extracted from the image data after the time information is encoded, and accurate image recognition can be performed based on these features.

[0260] Image processing and analysis modules include:

[0261] Object tracking: Based on the results of image recognition, the position and bounding box of the initial target are determined, and the Kalman filter algorithm is used to track the target;

[0262] The calculation formula of the Kalman filter algorithm is:

[0263]

[0264] in, is the state estimate (position, speed, etc.) of the target at time k, F is the state transfer matrix, B is the control matrix, and u k is the control input, wk is the process noise, z k is the observation value, H is the observation matrix, v k is the observation noise;

[0265] Behavior analysis: Extract the target's motion trajectory and behavior characteristics (such as speed, acceleration, direction change, etc.) from the object tracking results, classify the extracted behavior characteristics, and identify the target's behavior pattern (such as walking, running, staying, etc.);

[0266] The calculation formula for behavioral feature classification is:

[0267]

[0268] in, is the behavior classification result, W is the weight matrix, f behavior is the behavioral feature vector, b is the bias;

[0269] Scene segmentation: Perform scene segmentation on image data and label each part of the image;

[0270] The calculation formula for scene segmentation is:

[0271]

[0272] Scene analysis: Based on the results of scene segmentation, scene analysis is performed to extract elements in the scene (such as people, cars, buildings, etc.) and understand the semantic information of the scene;

[0273] Through the above steps, the image processing and analysis module can comprehensively process and analyze the results of image recognition, realize object tracking, behavior analysis and scene understanding, and provide support for applications such as intelligent monitoring and autonomous driving.

[0274] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention is limited to these examples. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0275] The present invention is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image processing system having an image sensor, characterized in that It includes an image sensor and preprocessing module, a time-coded imaging module, a feature extraction and recognition module, an image processing and analysis module, and a display and storage module, wherein; The image sensor and the preprocessing module capture image data and preprocess the received image data, the preprocessing including denoising, white balance and color correction; The time-coded imaging module encodes time information into the preprocessed image data to achieve the capture and analysis of fast-moving targets, specifically including: Time coding: During the image data capture process, the time information is encoded into each frame of image data, so that the image data contains timestamp information; Motion detection: connected with time coding, it detects moving targets in image data based on time coding information, distinguishes moving objects from static background, and reduces motion artifacts; High frame rate imaging: Achieve high frame rate image capture and support image sequences with high temporal resolution; Spatiotemporal reconstruction: Spatiotemporal reconstruction based on time coding information and high frame rate image data, providing three-dimensional motion trajectory reconstruction; The feature extraction and recognition module extracts image features, including edge features, texture features, and shape features, from the image data after the temporal information is encoded, and performs image recognition based on the extracted image features, including classification, target detection, and semantic segmentation; The image processing and analysis module processes and analyzes the results of image recognition, including object tracking, behavior analysis and scene understanding; The display and storage module is used to display the analysis results and store the analysis results in a storage device for subsequent call and analysis; The space-time reconstruction includes: Time series analysis: Use the timestamp information provided by time coding to perform time series analysis on high frame rate image data to determine the image data sequence at each time point; Motion vector calculation: Calculate the motion vector of each pixel based on the optical flow algorithm to obtain the motion information of the image; 3D coordinate reconstruction: Utilize multi-viewpoint stereo vision method to reconstruct 3D coordinates through high frame rate image data from multiple perspectives; Trajectory reconstruction: Combine the particle filter algorithm to smooth and predict the reconstructed three-dimensional trajectory.

2. The image processing system having an image sensor according to claim 1, characterized in that: The image sensor and preprocessing module include: Image capture: Use image sensor to capture image data, the image sensor is CMOS sensor; Image preprocessing: Preprocess the received image data, including: Denoising: Use an adaptive filtering algorithm to denoise the image data to reduce random noise in the image data. The calculation formula is: Among them, I filtered (x,y) is the denoised image value, I(x+i,y+j) is the pixel value of the original image, w(i,j) is the weight of the filter, and k is the size of the filter window; White balance: adjusts the color balance in the image through the gray world algorithm to correct the influence of the light source color temperature. The calculation formula is: Among them, R(x, y), G(x, y) and B(x, y) are the values ​​of the red, green and blue channels of the image at position (x, y), respectively. avg , G avg and B avg is the average value of the red, green, and blue channels in the image, and N is the total number of pixels in the image; Adjusted white balance value: Where R′(x, y) and B′(x, y) are the adjusted values ​​of the red and blue channels; Color correction: Use a color correction algorithm based on a lookup table to perform color correction on the image to correct the color deviation caused by changes in sensor characteristics or lighting conditions. The calculation formula is: R′=LUT R (R); G′=LUT G (G); B′=LUT B (B); Among them, R, G, and B are the values ​​of the red, green, and blue channels of the original image, respectively, and R', G', and B' are the values ​​of the red, green, and blue channels after color correction, respectively. R , LUT G and LUTs B They are lookup tables for the red, green, and blue channels respectively.

3. The image processing system having an image sensor according to claim 1, characterized in that: The time code includes: Timestamp generation: When the image sensor captures each frame of image data, the image processing system clock generates the corresponding timestamp information T i , where i represents the frame number, and the timestamp records the time when the corresponding frame image data is captured; Timestamp embedding: The timestamp information is encoded into the image frame through the Alpha channel. The calculation formula is: I′(x,y,α)={I(x,y),T i }; Among them, α represents the additional channel; Time synchronization: Ensure the synchronization of timestamp information and image data, and keep the clock of the image processing system consistent with the capture frequency of the image sensor to avoid offset and error of time information.

4. The image processing system having an image sensor according to claim 3, characterized in that: The motion detection includes: Time difference calculation: Use the timestamp information provided by the time code to perform time difference calculation on the image data of adjacent frames to obtain motion information. The calculation formula is: D t (x,y)=I(x,y,t)-I(x,y,t-Δt); Among them, I(x, y, t) and I(x, y, t-Δt) are the image frames with timestamps t and t-Δt, respectively. t (x, y) is the time difference result of the corresponding pixel position; Threshold segmentation: Perform threshold segmentation on the time difference result to distinguish the moving object from the static background. The calculation formula is: Wherein, τ is the preset threshold, M(x, y) is the binarized motion detection result, 1 indicates motion is detected, and 0 indicates stillness; Artifact removal: By using time coding information and analyzing the motion consistency between consecutive frames, artifacts caused by noise or illumination changes are eliminated. The calculation formula is: M'(x, y)=M(x, y)∩M(x, y, t+Δt); Wherein, M′(x, y) is the motion detection result after eliminating artifacts.

5. The image processing system having an image sensor according to claim 4, characterized in that: The high frame rate imaging comprises: High-speed image sensor: Use high-speed image sensor to capture high frame rate image data; Fast data readout: Transmit captured high frame rate image data via low voltage differential signal; Cache storage: Temporarily store captured high frame rate image data to avoid delays and losses during data transmission; Frame synchronization control: Through the clock synchronization mechanism, the time consistency of each frame of image data during capture and transmission is ensured to avoid time deviation between frames. The calculation formula is: t i+1 =t i +Δt; Among them, t i and t i+1 They represent the timestamps of the i-th frame and the i+1-th frame respectively, and Δt is the time interval between frames.

6. The image processing system having an image sensor according to claim 5, characterized in that: The time series analysis is expressed as: S(t)={I(x,y,t)|t=t0,t1,...,t n }; Where S(t) represents the image sequence corresponding to time t, and I(x, y, t) is the image data at time t; The motion vector calculation is expressed as: I x u+I y v+I t =0; Among them, I x is the partial derivative of image brightness in the x direction, I y is the partial derivative of image brightness in the y direction, I t is the partial derivative of the image brightness at time t, u is the velocity component of the optical flow in the x direction, and v is the velocity component of the optical flow in the y direction; The three-dimensional coordinate reconstruction is expressed as: Where n is the number of viewpoints, f i and B i are the focal length and baseline distance of the i-th viewpoint, respectively. and are the pixel coordinates in the ith viewpoint.

7. The image processing system having an image sensor according to claim 6, characterized in that: The particle filter algorithm includes: Initialization: Generate N particles and give the same initial weights The calculation formula for position initialization is: Among them, x0, y0, z0 are the initial positions, is the normally distributed noise, σ x , σ y , σ z is the standard deviation of the initial position noise; Particle prediction: predict particles based on the state transition model, combine time coding information and motion vector calculation results to update the state, including position prediction and motion vector update; The calculation formula for the position prediction is: in, is the predicted position of particle i at time k, is the predicted position of particle i at time k-1, is the motion vector component of the particle, Δt is the time interval; The calculation formula for the motion vector update is: Update weights: Update the weight of each particle according to the observation model, and adjust the weights based on the matching degree of time coding information and image data. The calculation formula is: in, is the probability of observing particle i at time k, z k is the actual observed value at time k, is the predicted observation value of particle i, σ z is the standard deviation of the observation noise; in, is the weight of particle i at time k, is the weight of particle i at time k-1; Normalized weight: Normalize the weight, the calculation formula is: in, is the normalized weight of particle i at time k; Resampling: Calculate the cumulative weight C based on the normalized weight j , generate a uniformly distributed random number u, and for each particle i, find j such that C j-1 <u<C j , select particle j as the new particle, and the calculation formula is: in, is a random number uniformly distributed in the interval [0,1]; State estimation: The state estimation of the image processing system is calculated based on the resampled particles, including position estimation and motion vector estimation; The calculation formula for the position estimation is: in, is the position estimate at time k; The calculation formula for the motion vector estimation is: in, is the motion vector estimate at time k.

8. The image processing system having an image sensor according to claim 1, characterized in that: The feature extraction and recognition module includes: Edge feature extraction: Use the Sobel operator to extract edge features from image data. The calculation formula is: Among them, G x and G y are the gradients of the image in the x and y directions respectively, and G is the edge strength; Texture feature extraction: Use the gray level co-occurrence matrix to extract the texture features of image data. The calculation formula is: Where P(i,j) is the probability of gray levels i and j appearing together in the gray level co-occurrence matrix; Shape feature extraction: Use Fourier descriptors to extract shape features of image data. The calculation formula is: Among them, F(u) is the Fourier descriptor, f(n) is the point on the image contour, N is the total number of contour points, j is the imaginary unit, and u is the frequency component; Image recognition: Use convolutional neural network models to perform image data classification, object detection, and semantic segmentation based on the extracted image features.

9. The image processing system having an image sensor according to claim 8, characterized in that: The convolutional neural network model includes: Convolution layer: Perform convolution operation on the input image data to extract low-level features. The calculation formula is: (I*K)(x,y)=∑ i ∑ j I(x-i,y-j)K(i,j); Where I is the input image, K is the convolution kernel, and (x, y) is the location of the output feature map; Pooling layer: Perform the maximum pooling operation on the convolutional feature map to reduce the size of the feature map. The calculation formula is: P(x,y)=max i,j I(x+i,y+j); Among them, P(x,y) is the feature map after pooling, and the max operation is to take the maximum value in the pooling window; Fully connected layer: Expand the pooled feature map into a one-dimensional vector, perform a fully connected operation, and output the classification result. The calculation formula is: in, is the classification result, W is the weight matrix, x is the input feature vector, and b is the bias; Loss function: The convolutional neural network model is trained using the cross entropy loss function, and the calculation formula is: Among them, L is the loss value, y i is the true label, is the predicted probability.

10. The image processing system having an image sensor according to claim 9, characterized in that: The image processing and analysis module comprises: Object tracking: Based on the results of image recognition, the position and bounding box of the initial target are determined, and the Kalman filter algorithm is used to track the target; Behavior analysis: extract the target's motion trajectory and behavior features from the object tracking results, classify the extracted behavior features, and identify the target's behavior pattern; Scene segmentation: Perform scene segmentation on image data and label each part of the image; Scene analysis: Based on the results of scene segmentation, scene analysis is performed to extract elements in the scene and understand the semantic information of the scene.

Images