Method and system for directly correcting fisheye image distortion based on video stream

Abstract

The invention discloses a fisheye image distortion direct correction method and system based on a video stream, and the method comprises the steps: obtaining a real-time video stream, and extracting an image of each frame from the obtained real-time video stream; performing equal-interval sampling on a curve structure corresponding to a linear structure in the real world in the fisheye image; performing ellipse fitting according to the sampling points, calculating to obtain an initial radius of a spherical correction model, and taking the center point of the fisheye image as the center point of the spherical correction model; according to the rule that the corrected sampling points are located on the same straight line, the mapping relation between the distance from the projection point on the target image to the center point of the target image and the radius of the corresponding spherical correction model is fitted; and according to the mapping relation, performing remapping operation on the captured fisheye image, moving each pixel point in the fisheye image to a corrected position to form a corrected panoramic image, and adding the corrected panoramic image to an output image queue for video file display. The method has the advantages of being simple in calibration process, high in real-time performance and low in calculation cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and system for directly correcting fisheye image distortion based on video stream. Background Art

[0002] In modern computer vision and image processing, fisheye lenses, due to their wide field of view (FOV), are widely used in various applications, such as autonomous driving, drone navigation, security monitoring, and virtual reality. However, this extremely large FOV is accompanied by significant nonlinear distortion, which leads to severe geometric distortion in the captured images, making them unsuitable for high-precision object detection and geometric measurement tasks. Therefore, correcting fisheye images has become a key research topic in computer vision.

[0003] Traditional fisheye correction methods rely heavily on complex calibration processes, estimating the distortion parameters of the fisheye lens by acquiring image data from calibration plates or specific structures. While these methods offer high accuracy, they often require extensive preliminary work, are complex to calibrate, and require a demanding experimental environment, making them difficult to adapt to applications requiring high real-time performance. On the other hand, calibration-free correction methods based on image content, while highly adaptable, suffer from high computational complexity and often struggle to meet real-time processing requirements. Summary of the Invention

[0004] The purpose of the present invention is to address the problems in the above-mentioned prior art and provide a method and system for direct correction of fisheye image distortion based on video stream, which has the advantages of simple calibration process, strong real-time performance and low computational cost, and is particularly suitable for real-time image correction tasks in dynamic scenes.

[0005] In order to achieve the above object, the present invention has the following technical solutions:

[0006] In a first aspect, a method for directly correcting fisheye image distortion based on a video stream is provided, comprising:

[0007] Acquire a real-time video stream and extract an image of each frame from the acquired real-time video stream;

[0008] On the extracted image, the curved structures corresponding to the real-world straight-line structures in the fisheye image are sampled at equal intervals. Ellipse fitting is performed based on the sampling points to calculate the initial radius of the spherical correction model, and the center point of the fisheye image is used as the center point of the spherical correction model. Based on the rule that the sampling points after correction are all located on the same straight line, the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius is fitted.

[0009] According to the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius, the captured fisheye image is remapped, and each pixel in the fisheye image is moved to the corrected position to form a corrected panoramic image, which is added to the output image queue for video file display.

[0010] As a preferred solution, the method of acquiring a real-time video stream and extracting an image of each frame from the acquired real-time video stream includes the following steps:

[0011] Initialize the connection to the fisheye camera. Log in to the fisheye camera by entering the correct device address, username, and password to obtain operation permissions. Configure the real-time video stream callback function to process the received data frames. Start real-time preview and set the data stream callback to trigger specific decoding and processing logic each time video stream data is received. Continuously monitor the video stream and use the callback function to decode and convert the format of each video frame, obtaining the image of each frame for distortion correction.

[0012] As a preferred solution, in the step of sampling the curved structures corresponding to the straight line structures in the fisheye image on the extracted image at equal intervals, the selected straight line structures are parallel to the imaging plane of the fisheye camera and perpendicular to the optical axis of the fisheye camera, and span the entire horizontal viewing angle of the fisheye image; the first point P0 is sampled at a position perpendicular to the horizontal diameter on the corresponding curved structure in the fisheye image, and sampling is performed at equal intervals to obtain a set of sampling points. Where n=20.

[0013] As a preferred solution, the step of performing ellipse fitting based on the sampling points to calculate the initial radius of the spherical correction model and taking the center point of the fisheye image as the center point of the spherical correction model is based on the obtained sampling point set. Perform ellipse fitting to calculate the initial radius r0 of the spherical correction model corresponding to point P0. The center point of the spherical correction model is the center point O(u0,v0) of the fisheye image.

[0014] As a preferred solution, the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius is fitted according to the rule that the corrected sampling points are all located on the same straight line, including the following steps:

[0015] Assume that the fisheye image is I, and the width and height of the fisheye image I are w respectively. f and h f ; The target image is I′, and the width and height of the target image I′ are w t and h t ; Sampling point P on the fisheye image i The corresponding projection point of (u,v) on the target image is Pi ′(x,y);

[0016] According to the collection The points in are all sampled from the corresponding arcs of the same straight line structure, so after correction, the corresponding projection points The y coordinates are the same, and the following relationship is obtained:

[0017] y0=y1=y i

[0018] y i =ρ i sinθ i

[0019] Where, ρ i is the point P on the target image i ′(x,y) to the center point O of the target image P (w t / 2,h t / 2) distance;

[0020]

[0021] Where d is the distance between the target image plane and the fisheye image plane, l i is the distance from the point on the fisheye image to the point in the center of the fisheye image, r i is the spherical radius of the corresponding spherical correction model;

[0022] Sampling point P i The radius r of the corresponding spherical correction model i The relationship between the initial radius r0 of the spherical correction model corresponding to the first sampling point P0 conforms to the following expression:

[0023]

[0024] According to the sampling point set Get a set of distances from the projection points on the target image to the center of the target image {ρ i ,i=1,2,...,n}, and the corresponding spherical correction model radius set {r i ,i=1,2,...,n},n=20.

[0025] As a preferred solution, the steps of performing a remapping operation on the captured fisheye image based on a mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius, moving each pixel in the fisheye image to a corrected position to form a corrected panoramic image, and adding the image to the output image queue for video file display include:

[0026] After a frame of fisheye image is captured from the video stream, the captured fisheye image is uploaded to the CPU memory; the functions provided by the Compute Unified Device Architecture (CUDA) module are used to transfer the original image data from the CPU memory to the GPU memory, and the parallel computing capability of the GPU is utilized to remap the captured fisheye image based on the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius; each pixel in the fisheye image is moved to the corrected position, and an interpolation method is used to ensure that the remapped image is smooth and undistorted, thereby eliminating the distortion of the fisheye image and restoring the captured fisheye image to a 180° panoramic image; the corrected panoramic image is downloaded from the GPU memory back to the CPU memory and added to the output image queue for video file display.

[0027] As a preferred solution, it also includes adopting a multi-threaded collaboration method to independently run the display thread and the video writing thread;

[0028] Each corrected image frame is added to the output image queue. A thread dedicated to display takes each frame from the output image queue and presents it in real time on the graphical interface. Another thread responsible for video writing tasks regularly obtains processed image frames from the output image queue, converts these image frames into video file format according to the set video encoding format and frame rate, saves them to the specified path, and generates a corrected panoramic image video file.

[0029] As a preferred solution, it also includes using mutex locks and conditional variables to manage image queues, using mutex locks to prevent multiple threads from modifying the same variable at the same time, creating three bidirectional queues to store image data at different stages, and using a graphic window to provide a visual interface to display the processed image; saving the processed video stream in a standard video file format to obtain a corrected panoramic image video file.

[0030] In a second aspect, a fisheye image distortion direct correction system based on video stream is provided, comprising:

[0031] An image extraction module is used to obtain a real-time video stream and extract an image of each frame from the obtained real-time video stream;

[0032] The sampling and fitting module is used to perform equally spaced sampling on the extracted image for the curved structures corresponding to the real-world straight line structures in the fisheye image; perform ellipse fitting based on the sampling points to calculate the initial radius of the spherical correction model, and use the center point of the fisheye image as the center point of the spherical correction model; and, based on the rule that the sampling points after correction are all located on the same straight line, fit the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius;

[0033] The remapping module is used to remap the captured fisheye image according to the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius, move each pixel in the fisheye image to the corrected position, form a corrected panoramic image, and add it to the output image queue for video file display.

[0034] According to a third aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in an electronic device to implement the method for directly correcting fisheye image distortion based on a video stream.

[0035] Compared with the prior art, the present invention has at least the following beneficial effects:

[0036] By extracting each frame of image from the acquired real-time video stream, since in the fisheye image, straight lines are bent into curves during the imaging process, by extracting the curved straight line structure in the fisheye image and sampling on the straight line, combined with a reasonable geometric model, the relationship between the distance from the projection point on the target image to the center point of the target image and the radius of the projection model can be directly calculated, thereby achieving efficient image correction. The present invention samples the curved structure corresponding to the straight line structure in the fisheye image of the real world at equal intervals on the extracted image, performs ellipse fitting based on the sampling points, calculates the initial radius of the spherical correction model, uses the center point of the fisheye image as the center point of the spherical correction model, and fits the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius based on the rule that the sampling points are all located on the same straight line after correction. Based on the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius, the captured fisheye image is remapped, and each pixel in the fisheye image is moved to the corrected position, thereby forming a corrected panoramic image. The correction method of the present invention has the advantages of simple calibration process, strong real-time performance and low computational cost, and is particularly suitable for real-time image correction tasks in dynamic scenes.

[0037] Furthermore, after capturing a frame of fisheye image from the video stream, the present invention uploads the captured fisheye image to the CPU memory of the central processing unit; uses the function provided by the Computing Unified Device Architecture CUDA module to transfer the original image data from the CPU memory to the GPU memory of the graphics processing unit; utilizes the parallel computing capability of the GPU to perform a remapping operation on the captured fisheye image based on the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius; downloads the corrected panoramic image from the GPU memory back to the CPU memory, and adds it to the output image queue for video file display. The correction method of the present invention is combined with GPU acceleration technology to achieve real-time processing on a computer. Utilizing the powerful parallel computing capability of the GPU greatly improves processing speed and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 Flowchart of a method for directly correcting fisheye image distortion based on video stream according to an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the linear structure sampling of a fisheye image according to an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of the fisheye image distortion correction principle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can also derive other embodiments without making any creative work.

[0043] See also Figure 1 The embodiment of the present invention provides a method for directly correcting fisheye image distortion based on a video stream, comprising the following steps:

[0044] S1, obtain real-time video stream, and extract each frame image from the obtained real-time video stream;

[0045] S2. On the extracted image, sample the curved structures corresponding to the real-world straight-line structures in the fisheye image at equal intervals; perform ellipse fitting based on the sampling points to calculate the initial radius of the spherical correction model, and use the center point of the fisheye image as the center point of the spherical correction model; based on the rule that the sampling points after correction are all located on the same straight line, fit the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius;

[0046] S3. Based on the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius, the captured fisheye image is remapped, and each pixel in the fisheye image is moved to the corrected position to form a corrected panoramic image, which is added to the output image queue for video file display.

[0047] In one possible implementation, the fisheye camera of an embodiment of the present invention uses a Hikvision camera. First, the connection to the Hikvision camera is initialized. Then, the correct device address, user name, and password are entered to log in to the camera device to obtain operation permissions. Then, the callback function of the real-time video stream is configured to process the received data frames. Then, the real-time preview is started and the data stream callback is set to trigger specific decoding and processing logic each time the video stream data is received. Finally, the video stream is continuously monitored and the callback function is used to decode and convert the format of each frame of video to obtain the image of each frame for distortion correction. This process ensures that the original video stream captured by the camera can be correctly received, decoded, and converted into an operable image format. Step S1 uses the SDK interface of the Hikvision camera software development kit to log in to the remote camera device through the IP address, port number, user name, and password. After obtaining the channel number of the playback library, setting the real-time video stream mode, and setting the decoding callback function, etc., the video stream starts to play and supports real-time preview.

[0048] In one possible implementation, see Figure 2 , select an arc structure in the fisheye image that is a straight line in the real world. This straight line structure is required to be parallel to the imaging plane of the fisheye camera and perpendicular to the optical axis of the fisheye camera. At the same time, it spans the entire horizontal viewing angle of the fisheye image. The first point P0 is sampled at a position perpendicular to the horizontal diameter of the corresponding curve structure in the fisheye image, and sampling is performed at equal intervals to obtain a set of sampling points Where n=20.

[0049] According to the obtained sampling point set Perform ellipse fitting to calculate the initial radius r0 of the spherical correction model corresponding to point P0. The center point of the spherical correction model is the center point O(u0,v0) of the fisheye image.

[0050] The direct fisheye image distortion correction method based on the video stream of the embodiment of the present invention can calculate the functional relationship r between the projection radius according to the rule that the sampling points are located on the same straight line after projection. i =f(r0); then according to The distance ρ from the projection point on the corresponding target image to the center point of the target image can be calculated i , where d is the distance between the target image plane and the fisheye image plane, l i is the distance from the point on the fisheye image to the midline point of the fisheye image; r is obtained by fitting i and ρ i The relationship between the fisheye image and the target image is finally generated.

[0051] In one possible implementation, see Figure 3 , let the fisheye image be I, the width and height of the fisheye image I are w f and h f ; The target image is I′, and the width and height of the target image I′ are w t and h t ; Sampling point P on the fisheye image i The corresponding projection point of (u,v) on the target image is P i ′(x,y); due to the set The points in are all sampled from the corresponding arcs of the same straight line structure (the straight line structure is parallel to the imaging plane of the fisheye camera and perpendicular to the optical axis of the fisheye camera). Therefore, after correction, the corresponding projection points The y coordinates are the same, and the following relationship is obtained:

[0052] y0=y1=y i

[0053] y i =ρ i sinθ i

[0054] Where, ρ i is the point P on the target image i ′(x,y) to the center point O of the target image P (w t / 2,h t / 2) distance;

[0055]

[0056] Where d is the distance between the target image plane and the fisheye image plane, l i is the distance from the point on the fisheye image to the point in the center of the fisheye image, r i is the spherical radius of the corresponding spherical correction model;

[0057] Combining the above formula, we can get the sampling point P i The radius r of the corresponding spherical correction model i The relationship between the initial radius r0 of the spherical correction model corresponding to the first sampling point P0 conforms to the following expression:

[0058]

[0059] Therefore, according to the sampling point set A set of distances {ρ i ,i=1,2,...,n}, and the corresponding spherical correction model radius {r i ,i=1,2,...,n},n=20.

[0060] Different from the traditional method using polynomial fitting, the mathematical model of the fitting function in the method of the present invention is The mathematical model is used to further calculate the mapping relationship between the fisheye image and the target image.

[0061] In one possible implementation, in step S3, after a frame of fisheye image is captured from the video stream, the captured fisheye image is uploaded to the CPU memory of the central processing unit (CPU); using the function provided by the CUDA (Compute Unified Device Architecture) module, the original image data is transferred from the CPU memory to the GPU memory of the graphics processing unit (GPU); utilizing the parallel computing capability of the GPU, the captured fisheye image is remapped according to the mapping relationship between the distance from the projection point on the target image to the center point of the target image obtained in step S2 and the corresponding spherical correction model radius. Utilizing the powerful parallel computing capability of the GPU, the processing speed and efficiency are greatly improved. Each pixel in the fisheye image is moved to the corrected position, and an interpolation method is used to ensure that the remapped image is smooth and undistorted, thereby eliminating the distortion of the fisheye image and restoring the captured fisheye image to a 180° panoramic image; the corrected panoramic image is downloaded from the GPU memory back to the CPU memory and added to the output image queue for video file display.

[0062] In one possible implementation, the method for direct correction of fisheye image distortion based on video stream in an embodiment of the present invention adopts a multi-threaded collaborative approach, in which the display thread and the video writing thread run independently; each corrected frame of the image is added to the output image queue, and a thread specifically responsible for display takes each frame of the image from the output image queue and presents it in real time on a graphical interface; another thread responsible for the video writing task periodically obtains processed image frames from the output image queue, and converts these image frames into a video file format according to the set video encoding format (such as MP4) and frame rate, saves them to a specified path, and finally generates a corrected panoramic image video file.

[0063] In one possible implementation, the method for direct correction of fisheye image distortion based on video streams in an embodiment of the present invention uses mutex locks and conditional variables to manage image queues, prevents multiple threads from modifying the same variable at the same time through mutex locks, creates three bidirectional queues to store image data at different stages, and uses a graphic window to provide a visual interface to display the processed image; the processed video stream is saved in a standard video file format to obtain a corrected panoramic image video file.

[0064] The direct fisheye image distortion correction method based on video stream in the embodiment of the present invention has the advantages of simple calibration process, strong real-time performance and low computational cost, and is particularly suitable for real-time image correction tasks in dynamic scenes.

[0065] Another embodiment of the present invention further provides a direct fisheye image distortion correction system based on video stream, comprising:

[0066] An image extraction module is used to obtain a real-time video stream and extract an image of each frame from the obtained real-time video stream;

[0067] The sampling and fitting module is used to perform equally spaced sampling on the extracted image for the curved structures corresponding to the real-world straight line structures in the fisheye image; perform ellipse fitting based on the sampling points to calculate the initial radius of the spherical correction model, and use the center point of the fisheye image as the center point of the spherical correction model; and, based on the rule that the sampling points after correction are all located on the same straight line, fit the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius;

[0068] The remapping module is used to remap the captured fisheye image according to the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius, move each pixel in the fisheye image to the corrected position, form a corrected panoramic image, and add it to the output image queue for video file display.

[0069] Another embodiment of the present invention further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the video stream-based direct fisheye image distortion correction method.

[0070] Exemplarily, the instructions stored in the memory may be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to implement the method for direct correction of fisheye image distortion based on a video stream according to an embodiment of the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the server.

[0071] The electronic device may be a computing device such as a smartphone, laptop, PDA, or cloud server. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the electronic device may include more or fewer components, or a combination of certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, and the like.

[0072] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0073] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk equipped on the server, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is about to be output.

[0074] It should be noted that the information interaction, execution process, etc. between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0075] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0076] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0077] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0078] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A direct fisheye image distortion correction method based on video stream, characterized in that: include: Acquire a real-time video stream and extract an image of each frame from the acquired real-time video stream; On the extracted image, the curved structures corresponding to the straight line structures in the real world in the fisheye image are sampled at equal intervals; Ellipse fitting is performed based on the sampling points to calculate the initial radius of the spherical correction model. The center point of the fisheye image is used as the center point of the spherical correction model. Based on the rule that all the sampling points after correction are located on the same straight line, the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius is fitted. According to the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius, the captured fisheye image is remapped, and each pixel in the fisheye image is moved to the corrected position to form a corrected panoramic image, which is added to the output image queue for video file display.

2. The method for directly correcting fisheye image distortion based on video stream according to claim 1, characterized in that: The method of acquiring a real-time video stream and extracting an image of each frame from the acquired real-time video stream comprises the following steps: Initialize the connection to the fisheye camera. Log in to the fisheye camera by entering the correct device address, username, and password to obtain operation permissions. Configure the real-time video stream callback function to process the received data frames. Start real-time preview and set the data stream callback to trigger specific decoding and processing logic each time video stream data is received. Continuously monitor the video stream and use the callback function to decode and convert the format of each video frame, obtaining the image of each frame for distortion correction.

3. The method for directly correcting fisheye image distortion based on video stream according to claim 1, characterized in that: In the step of sampling the curved structures corresponding to the straight line structures in the fisheye image at equal intervals on the extracted image, the selected straight line structures are parallel to the imaging plane of the fisheye camera and perpendicular to the optical axis of the fisheye camera, and span the entire horizontal viewing angle of the fisheye image; the first point P0 is sampled at a position perpendicular to the horizontal diameter on the corresponding curved structure in the fisheye image, and sampling is performed at equal intervals to obtain a set of sampling points. Where n=20.

4. The method for directly correcting fisheye image distortion based on video stream according to claim 3, characterized in that: The step of performing ellipse fitting based on the sampling points, calculating the initial radius of the spherical correction model, and taking the center point of the fisheye image as the center point of the spherical correction model is based on the obtained sampling point set. Perform ellipse fitting to calculate the initial radius r0 of the spherical correction model corresponding to point P0. The center point of the spherical correction model is the center point O(u0,v0) of the fisheye image.

5. The method for directly correcting fisheye image distortion based on video stream according to claim 4, characterized in that: The method of fitting the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius according to the rule that the corrected sampling points are all located on the same straight line includes the following steps: Assume that the fisheye image is I, and the width and height of the fisheye image I are w respectively. f and h f ; The target image is I′, and the width and height of the target image I′ are w t and h t ; Sampling point P on the fisheye image i The corresponding projection point of (u,v) on the target image is P i ′(x,y); According to the collection The points in are all sampled from the corresponding arcs of the same straight line structure, so after correction, the corresponding projection points The y coordinates are the same, and the following relationship is obtained: y0=y1=y i y i =ρ i sinth i Where, ρ i is the point P on the target image i ′(x,y) to the center point O of the target image P (w t / 2,h t / 2) distance; Where d is the distance between the target image plane and the fisheye image plane, l i is the distance from the point on the fisheye image to the point in the center of the fisheye image, r i is the spherical radius of the corresponding spherical correction model; Sampling point P i The radius r of the corresponding spherical correction model i The relationship between the initial radius r0 of the spherical correction model corresponding to the first sampling point P0 conforms to the following expression: According to the sampling point set Get a set of distances from the projection points on the target image to the center of the target image {ρ i ,i=1,2,...,n}, and the corresponding spherical correction model radius set {r i ,i=1,2,...,n},n=20.

6. The method for directly correcting fisheye image distortion based on video stream according to claim 1, characterized in that: The steps of performing a remapping operation on the captured fisheye image according to a mapping relationship between a distance from a projection point on the target image to a center point of the target image and a corresponding spherical correction model radius, moving each pixel in the fisheye image to a corrected position to form a corrected panoramic image, and adding the image to an output image queue for video file display include: After a frame of fisheye image is captured from the video stream, the captured fisheye image is uploaded to the CPU memory; the functions provided by the Compute Unified Device Architecture (CUDA) module are used to transfer the original image data from the CPU memory to the GPU memory, and the parallel computing capability of the GPU is utilized to remap the captured fisheye image based on the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius; each pixel in the fisheye image is moved to the corrected position, and an interpolation method is used to ensure that the remapped image is smooth and undistorted, thereby eliminating the distortion of the fisheye image and restoring the captured fisheye image to a 180° panoramic image; the corrected panoramic image is downloaded from the GPU memory back to the CPU memory and added to the output image queue for video file display.

7. The method for directly correcting fisheye image distortion based on video stream according to claim 1, characterized in that: It also includes the use of multi-threaded collaboration to allow the display thread and video writing thread to run independently; Each corrected image frame is added to the output image queue. A thread dedicated to display takes each frame from the output image queue and presents it in real time on the graphical interface. Another thread responsible for video writing tasks regularly obtains processed image frames from the output image queue, converts these image frames into video file format according to the set video encoding format and frame rate, saves them to the specified path, and generates a corrected panoramic image video file.

8. The method for directly correcting fisheye image distortion based on video stream according to claim 1, characterized in that: It also includes using mutex locks and conditional variables to manage image queues, using mutex locks to prevent multiple threads from modifying the same variable at the same time, creating three bidirectional queues to store image data at different stages, and using graphics windows to provide a visual interface to display the processed images; saving the processed video stream in a standard video file format to obtain a corrected panoramic image video file.

9. A fisheye image distortion direct correction system based on video stream, characterized in that: include: An image extraction module is used to obtain a real-time video stream and extract an image of each frame from the obtained real-time video stream; The sampling and fitting module is used to perform equally spaced sampling on the extracted image for the curved structures corresponding to the straight-line structures in the real world in the fisheye image; Ellipse fitting is performed based on the sampling points to calculate the initial radius of the spherical correction model. The center point of the fisheye image is used as the center point of the spherical correction model. Based on the rule that all the sampling points after correction are located on the same straight line, the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius is fitted. The remapping module is used to remap the captured fisheye image according to the mapping relationship between the distance from the projection point on the target image to the center point of the target image and the corresponding spherical correction model radius, move each pixel in the fisheye image to the corrected position, form a corrected panoramic image, and add it to the output image queue for video file display.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the direct correction method for fisheye image distortion based on video stream as described in any one of claims 1 to 8.

Images