Non-vision-field stable imaging method and system

Abstract

The invention discloses a non-vision-field stable imaging method and a non-vision-field stable imaging system. The non-vision-field stable imaging method comprises the following steps: acquiring transformation from a previous frame to a current frame of all frames in a video; accumulating and transforming to obtain an image track; smoothing the trajectory by using an average window; generating a new transformation matrix from the previous frame to the current frame; and applying the new transformation matrix to each frame of the video, and synthesizing the processed frame sequence into a final video. The system comprises a camera module, a processor module and a helmet module. According to the method, target identification is carried out in the vision field, hidden targets can be identified under the non-vision field condition, the exposure risk is remarkably reduced, the investigation efficiency and safety are improved, and the reliability of performance under different scenes can be ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of non-line-of-sight imaging, and in particular relates to a non-line-of-sight stable imaging method and system. Background Art

[0002] In counter-terrorism, search and rescue and other missions, users often need to rely on detection equipment for information collection and target identification when performing tasks in complex and dynamic environments. However, the use environment is usually accompanied by shock, bumps and unstable factors, such as uneven ground when traveling, vibrations caused by vehicle movement and bad weather. These external factors cause users to have significant control difficulties when using the equipment, affecting the stability of the equipment and imaging quality.

[0003] In order to solve the vibration and bump problems faced by users, some technical solutions have been applied to detection equipment. For example, some high-end imaging equipment has introduced electronic stabilization systems and mechanical shock absorbers, which achieve image stabilization by using sensors such as gyroscopes and accelerometers to ensure that relatively clear images can be obtained in dynamic environments. In addition, the application of shock-absorbing materials has also improved the shock resistance of the equipment to a certain extent, so that users can keep the equipment running normally during intense or unstable actions.

[0004] Although existing technologies have made some progress in dealing with vibrations and bumps, there are still significant limitations. First, electronic stabilization systems often have high power consumption requirements for equipment, which leads to reduced endurance of the equipment and affects its effectiveness in long-term use. Secondly, mechanical shock absorbers may not be able to adapt in real time when dealing with severe vibrations, resulting in blurred or jittery images. In addition, these technologies are not adaptable to the environment during use, especially in extreme weather conditions, and imaging stability cannot be guaranteed. Therefore, there is an urgent need for a new technology that can effectively improve the stability and imaging quality of detection equipment under shock and bump conditions, so as to enhance the user's information perception ability and ensure that reliable information can be obtained in complex environments. How to achieve higher-precision non-line-of-sight imaging and then achieve detailed perception of complex non-line-of-sight scenes is the core issue currently being addressed in the field of non-line-of-sight imaging. Summary of the invention

[0005] The present invention aims to provide a non-line-of-sight stable imaging method and system, which improves the efficiency and safety of detection and ensures reliability in different scenarios.

[0006] In order to achieve the purpose of the present invention, on the one hand, the present invention provides a non-viewing area stable imaging method, comprising the following steps:

[0007] Step 1: Get the transformation from the previous frame to the current frame of all frames in the video;

[0008] Step 2: Accumulate transformation to obtain image trajectory;

[0009] Step 3: Smooth the trajectory using an average window;

[0010] Step 4: Generate a new transformation matrix from the previous frame to the current frame;

[0011] Step 5: Apply the new transformation matrix to each frame of the video and synthesize the processed frame sequence into the final video.

[0012] In another aspect, the present invention provides a non-line-of-sight stable imaging system, the stable imaging system comprising the following modules:

[0013] A camera module, used to capture images of the user's surroundings to form video stream data;

[0014] A processor module is used to train the target detection model and identify targets in the non-viewing area in real time through machine learning algorithms, image recognition technology and deep learning methods;

[0015] The helmet module is used to detect jitter in the image and uses a motion estimation algorithm to determine the movement of the head by analyzing the movement of pixels between adjacent frames.

[0016] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a non-line-of-sight stable imaging method.

[0017] Compared with the prior art, the significant progress of the present invention lies in: (1) The present invention can effectively reduce the impact of jitter on the user during walking by introducing a shock-absorbing device and an intelligent algorithm, thereby achieving high image stability. This technical advantage ensures that the captured image remains clear in a dynamic environment, thereby improving the accuracy of information acquisition; (2) The present invention adopts a high-precision sensor combined with an efficient processing algorithm, which can analyze and correct image jitter in real time, enhance real-time stability, enable users to obtain reliable information during rapid movement, and improve real-time processing capabilities; (3) The present invention can reduce the power consumption of the device and extend the battery life while ensuring image stability by optimizing the design and energy efficiency management; (4) The design of the present invention takes into account the needs of a variety of complex application environments and can adapt to different terrains and climatic conditions. This adaptability enables users to ensure the stability and reliability of image acquisition under various extreme conditions.

[0018] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below in conjunction with the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 is a flow chart of the method proposed by the present invention;

[0021] Figure 2 It is the application scenario of the present invention;

[0022] Figure 3 It is a demonstration diagram of the present invention.

[0023] The reference numerals in the figure are: 1-wall; 2-receiving camera; 3-helmet. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0025] Combination Figure 1 and Figure 2 In one aspect, the present invention provides a non-viewing area stable imaging method, comprising the following steps:

[0026] Step 1: Get the transformation from the previous frame to the current frame of all frames in the video;

[0027] Step 2: Accumulate transformation to obtain image trajectory;

[0028] Step 3: Smooth the trajectory using an average window;

[0029] Step 4: Generate a new transformation matrix from the previous frame to the current frame;

[0030] Step 5: Apply the new transformation matrix to each frame of the video, and synthesize the processed frame sequence into the final video to ensure that there is no highlight during playback.

[0031] Furthermore, the step 1 comprises the following steps:

[0032] 1-1: Extract and optimize feature corner points: obtain and track feature corner points through corner detection, and use the corner detection algorithm to extract stable feature corner points from each frame; the feature corner points usually appear as edge intersections or areas with significant local intensity changes in the image. Among the feature corner points, the optimal feature points can be selected according to several thresholds to ensure the accuracy and efficiency of subsequent matching and calculation processes.

[0033] 1-2: Calculate motion difference: Calculate the difference between adjacent frames of the image through global optical flow and image matching; Calculate the motion of the feature corner points between adjacent frames using the optical flow method;

[0034] 1-3: Estimating the displacement and rotation matrix: Obtain the displacement and rotation matrix between the adjacent frames through matrix transformation; Calculate the transformation matrix using the RANSAC algorithm based on the matched feature corner points.

[0035] Furthermore, the optical flow method in step 1-2 is based on the assumption that: within a short period of time of a few seconds, the movement of the object is continuous, so the change in pixel intensity between adjacent frames is described by a motion vector; the motion vector is obtained by calculating the change in the corresponding position of each feature corner point in adjacent frames; the motion vector contains displacement information and further estimates the rotation component.

[0036] Furthermore, the RANSAC algorithm comprises the following steps:

[0037] (1) Randomly select samples: randomly select the minimum number of points from the data set;

[0038] (2) Model estimation: Based on the selected sample points, the model parameters (e.g., displacement matrix and rotation matrix) are calculated;

[0039] (3) Consistency evaluation: Calculate the fitness of the model in the entire data set and count the number of inliers that are consistent with the model;

[0040] (4) Model update: If the number of inliers in the current model is greater than the maximum number of inliers found previously, the optimal model parameters are updated;

[0041] (5) Iteration process: Repeat the above steps a preset number of times (or until a simple satisfaction condition is reached) to increase the chance of finding the best model;

[0042] (6) Output results: The model with the largest number of inliers is finally output, and the model parameters are recalculated to include more inliers.

[0043] Furthermore, the step 2 comprises the following steps:

[0044] 2-1: Initialization: Take the first frame of the video as the starting frame and set the unit change matrix T0, usually expressed as T0=I;

[0045] 2-2: Frame processing loop: Starting from the second frame, each frame i is processed in sequence starting from frame 1. For each frame i, the change matrix T from the previous frame i-1 to the current frame i is extracted i-1 →i;

[0046] 2-3: Replace the cumulative transformation matrix: Update the cumulative transformation matrix through matrix multiplication:

[0047] T i =T i-1 ·T i -1→i;

[0048] Among them, Ti is the cumulative transformation matrix from the starting frame to the current frame i;

[0049] 2-4: Extract displacement and rotation information: Extract the displacement vector (dx, dy) and the rotation angle da from the accumulated transformation matrix Ti; the displacement vector dx, dy is extracted from the last column of the transformation matrix, and the rotation angle da is obtained by solving the rotation part of the transformation matrix; the displacement vector (dx, dy) and the rotation angle da of each frame are stored in an array or a list to form a complete image transformation trajectory;

[0050] 2-5: Visualize the change trajectory: Visualize the stored displacement vector (dx, dy) and rotation angle da to generate a trajectory line representing the entire video frame transformation process. The trajectory line shows the overall trend and change of the camera movement, and obtains a data structure containing the transformation trajectory of all frame images, which provides basic data support for subsequent trajectory analysis and image stabilization processing.

[0051] Furthermore, the step 3 is specifically as follows:

[0052] Select the window size N according to the frame rate of the video and the jitter during the shooting process. The larger the window, the smoother it is.

[0053] Compute the smoothed displacement vector within the window:

[0054]

[0055] Among them, j is the index variable. When calculating dx smooth [i], j from Change to It means that i is the center and the left and right sides are expanded The trajectories of the positions are summed;

[0056] Compute the smoothed rotation angle within a window:

[0057]

[0058] The smooth transformation information of each frame is stored in an array.

[0059] Furthermore, the step 4 is specifically as follows:

[0060] 4-1: Generate rotation matrix: For each frame i, use the smooth displacement vector (dxsmooth ,dy smooth ) and the smoothed rotation vector da smooth Generate a new transformation matrix. First, the smooth rotation vector da smooth Generate the rotation matrix R:

[0061]

[0062] 4-2: Generate transformation matrix: Use the above rotation matrix R and add the smooth displacement vector (dx smooth ,dy smooth ), construct a new transformation matrix:

[0063]

[0064] For each frame i in the video, use a new transformation matrix to map the image of the previous frame i-1 to the current frame i to generate a stable image frame;

[0065] 4-3: Store the generated stable image frame sequence in a list.

[0066] Furthermore, the window size is 3 frames, 5 frames or 7 frames.

[0067] On the other hand, the present invention provides a non-viewing area stable imaging system, combined with Figure 3 , the stable imaging system comprises the following modules:

[0068] The camera module is used to capture images of the user's surroundings to form video stream data, especially focusing on capturing targets in non-viewing areas such as potential threats to the side or rear;

[0069] A processor module is used to train the target detection model and identify targets in the non-viewing area in real time through machine learning algorithms, image recognition technology and deep learning methods;

[0070] The helmet module is used to detect jitter in the image and uses a motion estimation algorithm to determine the movement of the head by analyzing the movement of pixels between adjacent frames.

[0071] The helmet device 3 and the camera 2 can provide important detection information. However, due to the unstable head movement of the user during movement, the image captured by the camera often shakes, affecting the effect of target recognition and subsequent processing. Therefore, it is very important to design an efficient algorithm to reduce the shake and stabilize the image.

[0072] The accumulated transformation matrix is ​​used to correct each frame of the image. The image is smoothed by interpolation for the detected jitter. A smoothing window (such as a weighted average of 3, 5, or 7 frames) is used to reduce random noise caused by motion and ensure stable imaging.

[0073] The specific implementation method of each module of the above system is the same as the aforementioned non-field of view stable imaging method, which will not be repeated here.

[0074] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0075] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A non-line-of-sight stable imaging method, characterized in that: The following steps are involved: Step 1: Get the transformation from the previous frame to the current frame of all frames in the video; Step 2: Accumulate transformation to obtain image trajectory; Step 3: Smooth the trajectory using an average window; Step 4: Generate a new transformation matrix from the previous frame to the current frame; Step 5: Apply the new transformation matrix to each frame of the video and synthesize the processed frame sequence into the final video.

2. A non-line-of-sight stable imaging method according to claim 1, characterized in that: The step 1 comprises the following steps: 1-1: Extract and optimize feature corner points: obtain tracking feature corner points through corner detection, and use the corner detection algorithm to extract stable feature corner points from each frame; 1-2: Calculate motion difference: Calculate the difference between adjacent frames of the image through global optical flow and image matching; Calculate the motion of the feature corner points between adjacent frames using the optical flow method; 1-3: Estimating the displacement and rotation matrix: Obtain the displacement and rotation matrix between the adjacent frames through matrix transformation; Calculate the transformation matrix using the RANSAC algorithm based on the matched feature corner points.

3. A non-line-of-sight stable imaging method according to claim 2, characterized in that: The optical flow method in step 1-2 is based on the assumption that the movement of the object is continuous within a short period of time of several seconds; the motion vector is obtained by calculating the change in the corresponding position of each feature corner point in adjacent frames.

4. The non-line-of-sight stable imaging method according to claim 2, characterized in that: The RANSAC algorithm includes the following steps: (1) Randomly select samples: randomly select the minimum number of points from the data set; (2) Model estimation: Calculate model parameters based on selected sample points; (3) Consistency evaluation: Calculate the fitness of the model in the entire data set and count the number of inliers that are consistent with the model; (4) Model update: If the number of inliers in the current model is greater than the maximum number of inliers found previously, the optimal model parameters are updated; (5) Repeat process: repeat the above steps for a preset number of times; (6) Output results: The model with the largest number of inliers is finally output and the model parameters are recalculated.

5. The non-line-of-sight stable imaging method according to claim 1, characterized in that: The step 2 comprises the following steps: 2-1: Initialization: Take the first frame of the video as the starting frame and set the unit change matrix T0, T0 = I; 2-2: Frame processing loop: Starting from the second frame, each frame i is processed in sequence starting from frame 1. For each frame i, the change matrix T from the previous frame i-1 to the current frame i is extracted i-1 →i; 2-3: Replace the cumulative transformation matrix: Update the cumulative transformation matrix through matrix multiplication: T i =T i-1 ·T i -1→i; Where Ti is the cumulative transformation matrix from the starting frame to the current frame i; 2-4: Extract displacement and rotation information: Extract the displacement vector (dx, dy) and rotation angle da from the accumulated transformation matrix Ti; store the displacement vector (dx, dy) and rotation angle da of each frame in an array or list to form an image transformation trajectory; 2-5: Visualize the change trajectory: Visualize the stored displacement vector (dx, dy) and rotation angle da, generate the trajectory line of the transformation process, and obtain a data structure containing the transformation trajectory of all frame images.

6. The non-line-of-sight stable imaging method according to claim 1, characterized in that: The step 3 is specifically as follows: Select the window size N according to the frame rate of the video and the jitter during shooting; Compute the smoothed displacement vector within the window: Among them, j is the index variable. When calculating dx smooth [i], j from Change to It means that i is the center and the left and right sides are expanded The trajectories of the positions are summed; Compute the smoothed rotation angle within a window: The smooth transformation information of each frame is stored in an array.

7. The non-line-of-sight stable imaging method according to claim 1, characterized in that: The step 4 is specifically as follows: 4-1: Generate rotation matrix: For each frame i, use the smooth displacement vector (dx smooth ,dy smooth ) and the smoothed rotation vector da smooth Generate a new transformation matrix. First, the smooth rotation vector da smooth Generate the rotation matrix R: 4-2: Generate transformation matrix: Use the above rotation matrix R and add the smooth displacement vector (dx smooth ,dy smooth ), construct a new transformation matrix: For each frame i in the video, use a new transformation matrix to map the image of the previous frame i-1 to the current frame i to generate a stable image frame; 4-3: Store the generated stable image frame sequence in a list.

8. The non-line-of-sight stable imaging method according to claim 6, characterized in that: The window size is 3 frames, 5 frames or 7 frames.

9. A non-line-of-sight stable imaging system, characterized in that: For implementing a non-line-of-sight stable imaging method as described in any one of claims 1 to 8, the stable imaging system comprises the following modules: A camera module, used to capture images of the user's surroundings to form video stream data; A processor module is used to train the target detection model and identify targets in the non-viewing area in real time through machine learning algorithms, image recognition technology and deep learning methods; The helmet module is used to detect jitter in the image and uses a motion estimation algorithm to determine the movement of the head by analyzing the movement of pixels between adjacent frames.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images