A panoramic image monitoring system, image processing method and computer equipment

Abstract

This invention discloses a panoramic image monitoring system, comprising: an image acquisition module for acquiring visual image data through an ultra-wide-angle image acquisition device with at least four sets of ring-shaped first devices; a data processing module for sequentially correcting and stitching the acquired image data to obtain panoramic image data, then performing target detection, recognition, and tracking operations on the processed image, and employing linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of the target; and a display module for displaying the processed image data and the detection, recognition, and tracking results of the target. This invention solves the problem of potential obstruction of the camera's field of view or severe degradation of image quality, effectively improving the system's stability and reliability. It also enables precise tracking of object motion, improving the accuracy and continuity of dynamic target tracking, and facilitating effective handling of situations where multiple targets move simultaneously or change rapidly.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, specifically to a real-time monitoring system for panoramic vehicle images, a method for processing panoramic vehicle images, computer equipment, and storage media. Background Technology

[0002] In modern vehicle safety systems, panoramic image monitoring systems have become one of the key technologies for improving driving safety. For example, in the prior art, there is a method, system and vehicle for generating panoramic images of a vehicle with patent application number CN201810380394.6, or a method, device and vehicle for generating panoramic surround view images with patent application number CN202411064436.7.

[0003] Panoramic imaging monitoring systems typically include cameras mounted around the vehicle to capture real-time images of the surrounding environment. These images are then processed using specific algorithms to provide the driver with clear and comprehensive visual information about the vehicle's surroundings. By utilizing high-resolution cameras and advanced image processing algorithms, the driver can gain a more intuitive understanding of the conditions around the vehicle, thereby avoiding potential collision risks.

[0004] Conventional panoramic image monitoring systems are mainly used for dynamic target tracking and scene recognition, such as the detection of moving objects like pedestrians and vehicles. However, in practical applications, when multiple dynamic targets appear simultaneously or move rapidly, the tracking ability of such systems is generally limited, making it impossible to accurately and continuously track each target. Secondly, in special environments such as snow, muddy roads, heavy rain, or dense fog, the camera's field of view may be obstructed or the image quality may be severely degraded, leading to reduced system performance or even failure. At the same time, conventional panoramic image monitoring systems also have limited ability to identify sharp metal objects or other dangerous obstacles, making it difficult to effectively warn and guide drivers to take evasive action. In addition, their ability to prevent "ghost peeks" (i.e., pedestrians suddenly appearing from blind spots) is also weak.

[0005] Therefore, this application proposes a panoramic image monitoring system, an image processing method, and a computer device to solve the above-mentioned technical problems. Summary of the Invention

[0006] The main objective of this invention is to provide a panoramic image monitoring system, image processing method, and computer equipment. By integrating a 360° no-dead-angle camera, a large visual screen, and advanced algorithms, it not only enhances the ability to identify various obstacles but also integrates other sensing devices. It can detect pedestrians' whereabouts in advance and automatically trigger emergency braking measures based on vehicle speed and distance, thereby effectively preventing traffic accidents caused by sudden situations and solving the technical problems mentioned in the background art.

[0007] The present invention solves the above-mentioned technical problems by adopting the following technical solutions:

[0008] A panoramic image monitoring system includes:

[0009] The image acquisition module is used to acquire visual image data through an ultra-wide-angle image acquisition device with at least four sets of rings of the first device;

[0010] The data processing module is used to sequentially correct and stitch the acquired image data to obtain panoramic image data. Then, it performs detection, recognition and tracking operations on the processed image for the specified target, and uses linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of the specified target.

[0011] The display module is used to display the processed image data and the detection, recognition and tracking results of the specified target.

[0012] Preferably, the data processing module further includes:

[0013] The preprocessing unit is used to perform preprocessing operations, including noise reduction, brightness adjustment, and contrast enhancement, on the acquired image data in sequence.

[0014] The distortion correction unit is used to determine the image distortion part of the preprocessed image data using image calibration technology, and to perform distortion correction operation according to the equipment parameters of the ultra-wide-angle image acquisition equipment.

[0015] The image stitching unit is used to identify overlapping areas and feature points in the image, and to seamlessly stitch together multiple distortion-corrected images using an image stitching algorithm.

[0016] The target detection, recognition and tracking unit is used to perform the detection, recognition and tracking of specified targets on image data, and uses linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of specified targets after image stitching.

[0017] Preferably, the target detection, recognition, and tracking unit performs the following operations for detecting, recognizing, and tracking a specified target:

[0018] The trained deep learning network model is converted into a format suitable for embedded systems, and forward propagation is performed using model inference libraries including TensorFlowLite and OpenCV.

[0019] The algorithm threshold was adjusted and historical tracking was used to improve prediction accuracy. The algorithm performance was also verified based on ground truth data.

[0020] Preferably, the specific operation process of the image stitching algorithm for seamless image stitching includes:

[0021] L1. Feature points are detected in the image using a feature point detection algorithm and described as vectors using a feature descriptor algorithm. Matching point pairs are found by comparing feature point descriptions.

[0022] L2. Use the RANSAC algorithm to remove mismatched points;

[0023] L3. Through affine transformation or projection transformation, feature points in different images are mapped to the same coordinate system for image registration.

[0024] L4. Stitch multiple images onto the same canvas and perform overlay, blending, or completion processing.

[0025] Preferred options also include:

[0026] Sensor devices installed on the first device are used to accurately measure the distance between the first device and surrounding objects, and to mark the position of the objects on the panoramic image data;

[0027] The sensor device collects data to locate and calculate the actual coordinates of a specified feature point. The midpoint coordinates are calculated by comparing the actual coordinates of the specified feature point in the mapped coordinate system with the recognition coordinates in the mapped coordinate system identified by the image stitching unit. The midpoint coordinates are then used as the precise coordinates of the feature point in the mapped coordinate system for the specified target detection, recognition, and tracking processing of the target detection and tracking unit.

[0028] Preferably, a panoramic image processing method, based on any of the panoramic image monitoring systems described above, performs embedded data calculation and processing, including the following steps:

[0029] S1. For the acquired image data, preprocessing operations including noise reduction, brightness and contrast adjustment are performed on each frame of the image in sequence.

[0030] S2. After preprocessing, the image data is distorted according to the pre-calibrated acquisition equipment parameters. The distorted image data is then seamlessly stitched together to form a complete panoramic image data.

[0031] S3. Construct a specified coordinate system and use image acquisition equipment to process feature point vector data in order to perform object recognition and tracking operations.

[0032] Preferably, the specific operation flow for performing object recognition and tracking in step S3 includes:

[0033] S31. With the geometric center of the first device as the origin, the forward and backward movement direction of the first device as the x-axis, the left and right movement direction as the y-axis, and the direction perpendicular to the x-axis and y-axis as the z-axis, construct the device body coordinate system O-xyz;

[0034] S32. Feature point vector data processing: For the acquired image data, feature points are extracted using feature point detection algorithms including SIFT, SURF, and ORB;

[0035] S33. Let P be a specified feature point in the image acquired by the i-th acquisition device. i (x i y i ), calculate the vector of the feature point relative to the camera coordinate system. By including the rotation matrix R i Translation vector T i The extrinsic parameter matrix of the acquisition device transforms the feature point vector from the acquisition device coordinate system to the device body coordinate system. The transformation formula is as follows:

[0036]

[0037] Among them, (X) world Y world Z world ) represents the coordinates of the feature point in the device body coordinate system.

[0038] S34. Compare the changes in the feature point coordinates of the same object in the device body coordinate system at different times to achieve object tracking. At this time, use linear interpolation algorithm or curve fitting algorithm to improve the detection and recognition accuracy of the specified target.

[0039] Preferably, the process of using the linear interpolation algorithm in step S34 includes:

[0040] Select a specified feature point P(x, y) in the stitched image data. For two adjacent pixels P(x1, y1) and P(x2, y2) of this feature point, their pixel values ​​are I1 and I2, respectively.

[0041] The pixel value I of a specified feature point P is calculated using the linear interpolation formula:

[0042]

[0043] The pixel value of each feature point in the stitched image is calculated using a linear interpolation formula to smooth the image and reduce discontinuities caused by distortion.

[0044] Preferably, the curve fitting algorithm in step S34 includes the following steps:

[0045] Select any set of discrete feature points (x) from the stitched image data. i y i Let i = 1, 2, 3, ..., n, and use the quadratic polynomial y = ax 2The +bx+c algorithm fits the feature points of the image data to optimize image quality. The specific values ​​of the coefficients a, b, and c are determined using the least squares method to minimize the sum of squared errors. The smallest is:

[0046]

[0047] After obtaining the coefficients a, b, and c based on the above formula, the quadratic polynomial y = ax² can be used. 2 +bx+c is used to fit the feature points of the image data.

[0048] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0049] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0050] As can be seen from the above technical solution, the present invention provides a panoramic image monitoring system and an image processing method. Compared with the prior art, the present invention has the following advantages:

[0051] 1. This invention integrates a deep learning network model into the target detection, recognition, and tracking unit, and utilizes model inference libraries such as TensorFlow Lite and OpenCV to perform forward propagation. This enables the algorithm to automatically adjust the threshold based on real-time traffic conditions and improve prediction accuracy by leveraging historical tracking data.

[0052] 2. This invention corrects the feature point coordinate recognition results and algorithm performance based on ground truth data, ensuring the high accuracy and reliability of the system. Ultimately, it can effectively prevent the "ghost peek" phenomenon and avoid the occurrence of sudden traffic accidents.

[0053] 3. The present invention assigns priorities to different tasks based on the task scheduling mechanism of RTOS, ensuring that time-sensitive tasks are completed on time, and realizes synchronization and communication between tasks through mechanisms such as semaphores and mutexes, thereby ensuring the efficient operation and stability of the entire system.

[0054] 4. This invention improves the detection and recognition accuracy of specified targets by employing linear interpolation or curve fitting algorithms. It can maintain high image quality and accuracy even when the image is affected by environmental factors. It not only solves the problem that the camera's field of view may be obstructed or the image quality may be severely degraded, but also effectively improves the stability and reliability of the system, ensuring that the image monitoring system can work normally under various adverse weather conditions and enhancing driving safety.

[0055] 5. By setting a feature point tracking mechanism in the device's coordinate system, this invention can compare the changes in the feature point coordinates of the same object at different times, thereby achieving accurate tracking of the object's motion state. This not only improves the accuracy and continuity of dynamic target tracking, but also effectively copes with situations where multiple targets move simultaneously or change rapidly. Finally, it significantly enhances the system's adaptability to complex traffic environments and ensures driving safety.

[0056] 6. This invention utilizes a trained deep learning network model to classify and identify feature point data mapped to the device's body coordinate system. It can determine the object category and take corresponding warning or avoidance measures based on the category. This not only improves the accuracy of object recognition and reduces false alarms and missed alarms, but also allows for real-time adjustment of driving strategies based on the recognition results, significantly reducing the risk of traffic accidents caused by failure to detect or correctly identify obstacles in a timely manner.

[0057] It should be understood that the descriptions in this section are not intended to identify key or essential features of embodiments of the invention, nor are they intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Of course, implementing any product of the invention does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description

[0058] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0059] Figure 1 This is a schematic diagram of the module framework of the system of the present invention;

[0060] Figure 2 This is a schematic diagram illustrating the operation flow of image correction and stitching and target recognition and tracking in the method of the present invention;

[0061] Figure 3 This is a schematic diagram of the image correction and stitching operation process during the use of the system of the present invention;

[0062] Figure 4 This is a schematic diagram illustrating the target tracking and playback operation process of the system of the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] For details in the embodiments, please refer to Figures 1 to 4 .

[0065] like Figure 1 As shown in the embodiment of the present invention, a panoramic image monitoring system is proposed for monitoring and target recognition and tracking of vehicle panoramic images, including:

[0066] (1) Image acquisition module, responsible for acquiring image information around the vehicle. It is used to acquire visual image data through at least four sets of ultra-wide-angle cameras around the vehicle, which can realize the acquisition of all-round visual data. Generally, the cameras are installed in the front, rear, left and right positions of the vehicle. Its main function is to capture and acquire 360° visual information data around the vehicle so as to achieve the effect of monitoring the surrounding environment of the vehicle without blind spots, significantly improving the driver's ability to recognize the surrounding situation and enhancing driving safety.

[0067] It should be noted that the ultra-wide-angle camera used in this application is generally a 180-degree or 190-degree ultra-wide-angle fisheye camera, which has the characteristics of high resolution, high dynamic range and good low-light performance, ensuring that clear images can be captured under different lighting conditions. In addition, the wide-angle lens design can cover most of the area around the vehicle, and the camera design is small to reduce the impact on the vehicle's wind resistance.

[0068] (2) Sensor devices installed on the vehicle are used to accurately measure the distance between the vehicle and surrounding objects and to mark the position of objects on panoramic image data.

[0069] In practical use, sensor devices are arranged around the vehicle. The system can integrate other types of sensors such as ultrasonic sensors, radar, or lidar (LiDAR). These sensors are used to accurately measure the distance between the vehicle and surrounding objects and mark the positions of objects on a panoramic view. The data is crucial for autonomous driving and assisted parking functions.

[0070] Therefore, the data collected by the sensor device here is used to locate and calculate the actual coordinates of the specified feature point. The midpoint coordinates are calculated by comparing the actual coordinates of the specified feature point in the mapped coordinate system with the recognition coordinates in the mapped coordinate system identified by the image stitching unit. The midpoint coordinates are then used as the precise coordinates of the feature point in the mapped coordinate system for the specified target detection, recognition and tracking processing of the target detection and tracking unit.

[0071] (3) A data processing module, used to sequentially correct and stitch the acquired image data to obtain panoramic image data, and then perform target detection, recognition and tracking operations on the processed image, and use linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of the target. The data processing module is internally equipped with:

[0072] (a) Preprocessing unit, used to perform preliminary processing on the acquired raw image, including noise reduction, brightness adjustment, contrast enhancement and other operations, to improve image quality and provide a better foundation for subsequent processing;

[0073] (b) Distortion correction unit: Since ultra-wide-angle cameras produce image distortion, this unit is used to determine the image distortion part of the image data after preprocessing using image calibration technology, and to perform distortion correction operation according to the device parameters of ultra-wide-angle cameras, so that the shape and position of objects in the image are closer to the real situation.

[0074] (c) Image stitching unit, which is used to identify overlapping areas and feature points in the images and use image stitching algorithms to seamlessly stitch multiple distortion-corrected images to form a complete panoramic image of the vehicle's surroundings.

[0075] The specific operational process of seamlessly stitching images using the image stitching algorithm includes:

[0076] L1. Feature points are detected in the image using feature point detection algorithms (such as SIFT, SURF, ORB, etc.), and they are described as vectors using feature descriptor algorithms. At the same time, matching point pairs are found by comparing feature point descriptions using algorithms such as FLANN or brute-force matching.

[0077] L2. Use the RANSAC algorithm to remove mismatched points;

[0078] L3. Through affine transformation or projection transformation (perspective transformation), feature points in different images are mapped to the same coordinate system for image registration.

[0079] L4. Stitch multiple images onto the same canvas and perform overlay fusion or completion processing to achieve high-precision and robust image stitching.

[0080] (d) Target detection, recognition and tracking unit, which uses deep learning and other technologies to detect, recognize and track specified targets such as pedestrians, vehicles, obstacles and road signs in panoramic image data, providing drivers with richer environmental information. In addition, this unit also uses linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of specified targets after image stitching.

[0081] The target detection, recognition, and tracking unit's operation process for detecting, recognizing, and tracking specified targets includes: converting the trained machine learning model (such as a deep learning network) into a format suitable for embedded systems; performing forward propagation using a model inference library (such as TensorFlow Lite or OpenCV); then adjusting the algorithm threshold, improving prediction accuracy by utilizing tracking history, and verifying algorithm performance based on ground truth data. It can automatically adjust the algorithm threshold according to real-time traffic conditions and improve prediction accuracy by utilizing historical tracking data, ensuring high accuracy while running in real time and reducing false detections and missed detections.

[0082] In summary, during practical use, this data processing module performs corresponding data processing operations based on a high-performance embedded computing platform. This includes measures such as using memory pools to avoid fragmentation, rationally allocating processor resources to prevent CPU overload, and using dynamic power management technology to reduce energy consumption.

[0083] This embedded computing platform coordinates image acquisition from various cameras, processes image data, and stitches multiple images into a seamless panoramic image in real time. It also integrates image preprocessing functions, such as noise removal and contrast adjustment, and can process auxiliary data from sensors, such as distance information, thereby enhancing the accuracy and reliability of the system.

[0084] Furthermore, during use, thanks to the preprocessing unit, distortion correction unit, image stitching unit, and target detection, recognition, and tracking unit, image noise can be effectively removed, brightness and contrast adjusted, and precise distortion correction and seamless stitching can be performed, thereby improving image quality. The use of linear interpolation or curve fitting algorithms further enhances the detection and recognition accuracy of the specified target, and finally ensures that dynamic targets can still be accurately tracked and identified in complex environments, reducing the risk of accidents.

[0085] At this point, the embedded computing platform can precisely control the program's memory usage in terms of resource management and optimization, and use memory pools to avoid fragmentation; it can also rationally allocate processor resources to ensure that critical tasks are processed in a timely manner and avoid CPU overload; it can use dynamic power management technology to reduce energy consumption; and it also involves cross-compilation, using the maintenance of cross-compilation toolchains to ensure version compatibility of compilers, linkers and libraries, adapting to the target platform architecture and instruction set; and it can also use simulators for preliminary testing and debugging.

[0086] In addition, regarding the hardware component, it can be added that the image processor is the core hardware of this module, responsible for rapidly processing the large amount of image data captured by the camera, including complex calculations such as distortion correction, image stitching, and target detection. It typically employs a high-performance digital signal processor or a dedicated image processor chip.

[0087] (4) Display module, which is used to display the processed panoramic image data and the detection, recognition and tracking results of the specified target on the display screen in the vehicle. It usually supports 2D or 3D display mode. The driver can switch different perspectives as needed, such as bird's-eye view, front view, rear view, side view, etc.

[0088] It can also be added that the display used in this module is installed on the center console or dashboard inside the vehicle to display the panoramic image and related information processed by the system. It is generally a high-definition, high-contrast LCD screen, and the screen is mostly a touch screen, which facilitates the driver's interaction with the system. For example, the driver can select to view a specific view from a particular camera or adjust system settings. The high resolution and fast response time of the display screen can ensure a clear and lag-free view.

[0089] When the display module is interactive, in order to ensure the system responds in real time, the task scheduling, interrupt handling and time management mechanisms of the RTOS are also used in the development process to assign priorities to different tasks, ensure that time-sensitive tasks are completed on time, and at the same time realize the synchronization and communication between tasks through mechanisms such as semaphores, mutexes and message queues, coordinate data sharing and execution order, thereby ensuring the efficient operation and stability of the entire system.

[0090] Furthermore, in the software interface of the display module, the layout follows the principle of clarity and ease of understanding. Commonly used functions are set with simple operation methods, such as switching views through simple gestures or buttons. The size and color contrast of interface elements are reasonable, facilitating quick identification and reducing driver distraction. At the same time, the interface can automatically adjust brightness and color scheme according to ambient light, and provides feedback mechanisms such as touch feedback vibration or audio prompts for successful operation.

[0091] At this point, the interaction logic closely resembles natural language and intuitive operation, offering voice command recognition or large touch buttons for easy driver activation of functions. The system possesses context awareness capabilities, providing intelligent prompts based on driving status and environmental conditions, such as automatically activating parking assist lines and providing voice guidance during low-speed parking.

[0092] (5) Recording and playback module: It can record panoramic images during vehicle driving and store them in the vehicle storage device. When an accident occurs or when it is necessary to view historical images, they can be played back.

[0093] It can also be added that the storage devices used in this module are generally solid-state drives or high-capacity flash memory chips, used to store the programs, configuration files and recorded panoramic image data required for system operation.

[0094] Furthermore, by integrating a 360° all-around camera, a large visual screen, and advanced algorithms, and especially by adding an alarm device to identify sharp metal and non-metal objects, the system can issue a voice alarm to guide drivers to avoid danger when approaching sharp objects. At the same time, through integration with infrared and other sensing devices, it can know the whereabouts of pedestrians in advance and take emergency measures, thereby avoiding collisions caused by suddenly appearing pedestrians, significantly reducing traffic accidents caused by poor road conditions or unexpected obstacles, and improving driving safety.

[0095] On the other hand, the present invention also discloses a panoramic image processing method, which performs embedded data calculation and processing based on the panoramic image monitoring system proposed in the above embodiments, with reference to... Figures 2 to 4 As shown, the overall steps in actual use include:

[0096] Step 1: After the vehicle starts, the system software first performs initialization operations, including self-checking the hardware devices, checking whether the camera, image processor and other devices are working properly; loading the system configuration file, obtaining the camera parameters, image stitching algorithm parameters, etc.

[0097] Step 2: The image acquisition module controls each camera to start working and acquires image data around the vehicle at a certain frame rate. This image data is transmitted to the image processor in the form of a video stream through the connection cable.

[0098] Step 3: After receiving the acquired image data, the image processor performs preprocessing operations, including noise reduction, brightness and contrast adjustment, on each frame of the image to improve image quality and reduce the impact of noise and uneven lighting on subsequent processing.

[0099] Step 4: After preprocessing, the image data is distorted according to the pre-calibrated acquisition equipment parameters. The distorted image data from different cameras are then seamlessly stitched together using a stitching algorithm to form a complete panoramic image of the vehicle's surroundings.

[0100] Step 5: Construct a specified coordinate system and process feature point vector data using the camera to perform object recognition and tracking operations. The specific operation process for performing object recognition and tracking operations at this stage includes:

[0101] (1) With the geometric center of the vehicle as the origin, the forward and backward movement of the vehicle as the x-axis, the left and right movement as the y-axis, and the direction perpendicular to the x-axis and y-axis as the z-axis, construct the equipment body coordinate system O-xyz;

[0102] (2) Feature point vector data processing: For the acquired image data, feature points are extracted using feature point detection algorithms including SIFT, SURF, and ORB.

[0103] (3) Let P be a specified feature point in the image acquired by the i-th acquisition device. i (x i y i ), calculate the vector of the feature point relative to the camera coordinate system. By including the rotation matrix R i Translation vector T i The extrinsic parameter matrix of the acquisition device transforms the feature point vector from the acquisition device coordinate system to the device body coordinate system. The transformation formula is as follows:

[0104]

[0105] Among them, (X) world Y world Z world ) represents the coordinates of the feature point in the device body coordinate system.

[0106] (4) Compare the changes in the coordinates of the feature points of the same object in the coordinate system of the device body at different times to achieve object tracking. For example, if the coordinates of a feature point on an object in the coordinate system of the device body are (X1, Y1, Z1) at time t1, and the coordinates of the feature point change to (X2, Y2, Z2) at time t2, then the displacements of the object in the x, y, and z directions are ΔX = X2 - X1, ΔY = Y2 - Y1, and ΔZ = Z2 - Z1, respectively. The motion state of the object can be analyzed based on these displacement information.

[0107] By setting a feature point tracking mechanism in the device's coordinate system, the changes in the feature point coordinates of the same object at different times can be compared, thereby achieving accurate tracking of the object's motion state. This not only improves the accuracy and continuity of dynamic target tracking but also effectively addresses situations where multiple targets move simultaneously or change rapidly. Ultimately, it significantly enhances the system's adaptability to complex traffic environments and ensures driving safety.

[0108] In addition, for object recognition, trained machine learning models (such as deep learning networks) can be used to classify and identify feature point data mapped to the device's body coordinate system to determine the category of the object (such as pedestrians, vehicles, sharp objects, etc.).

[0109] Furthermore, in the process of detecting and recognizing a specified target based on feature points, linear interpolation algorithms or curve fitting algorithms can be used to improve the detection and recognition accuracy of the specified target. The usage process of the linear interpolation algorithm includes:

[0110] Select a specified feature point P(x, y) in the stitched image data. For two adjacent pixels P(x1, y1) and P(x2, y2) of this feature point, their pixel values ​​are I1 and I2, respectively.

[0111] The pixel value I of a specified feature point P is calculated using the linear interpolation formula:

[0112]

[0113] The pixel value of each feature point in the stitched image is calculated based on the linear interpolation formula to smooth the image and reduce discontinuities caused by distortion.

[0114] The workflow for using the curve fitting algorithm includes:

[0115] Select any set of discrete feature points (x) from the stitched image data. i y i Let i = 1, 2, 3, ..., n, and use the quadratic polynomial y = ax 2 The +bx+c algorithm fits the feature points of the image data to optimize image quality. The specific values ​​of the coefficients a, b, and c are determined using the least squares method to minimize the sum of squared errors. The smallest is:

[0116]

[0117] After obtaining the coefficients a, b, and c based on the above formula, the quadratic polynomial y = ax² can be used. 2 The +bx+c algorithm fits the feature points of the image data to adjust pixel values, optimize image quality, and improve the accuracy of object recognition and the determination of sharp objects. In addition, in the specific implementation process, this curve fitting algorithm can better adapt to complex image distortion situations and more effectively reduce distortion correction errors compared with linear interpolation.

[0118] In summary, using linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of specified targets can maintain high image quality and accuracy even when images are affected by environmental factors. This not only solves the problem of camera field of view being obstructed or image quality being severely degraded, but also effectively improves the stability and reliability of the system, ensuring that the image monitoring system can work normally under various adverse weather conditions and enhancing driving safety.

[0119] Furthermore, the feature point coordinate recognition results and algorithm performance can be corrected based on ground truth data, ensuring the system's high accuracy and reliability. Ultimately, this effectively prevents the "ghost peek" phenomenon and avoids sudden traffic accidents.

[0120] Step 6: Utilize deep learning algorithms to perform target detection and recognition on the stitched panoramic image, identifying pedestrians, vehicles, obstacles, road signs, and other targets in the image, and locating and tracking them. The detected target information is then fused with the panoramic image to provide the driver with more intuitive environmental information.

[0121] The aforementioned method utilizes a trained deep learning network model to classify and identify feature point data mapped to the device's body coordinate system. This enables the determination of object categories and the implementation of corresponding warnings or avoidance measures based on these categories. This not only improves the accuracy of object recognition and reduces false alarms and missed alarms, but also allows for real-time adjustments to driving strategies based on the recognition results, significantly reducing the risk of traffic accidents caused by the failure to detect or correctly identify obstacles in a timely manner.

[0122] Step 7: The processed panoramic image and object detection results are sent to the display module for display on the in-vehicle monitor. The driver can switch between different display perspectives to view detailed information about the vehicle's surroundings using the vehicle's control buttons or touchscreen. Simultaneously, the system can automatically switch to the appropriate perspective based on the vehicle's driving status, such as reversing or turning.

[0123] Step 8: The recording and playback module records the panoramic images in real time according to the set rules during vehicle operation and stores the image data in the vehicle's onboard storage device. When the driver needs to view historical images, they can select the corresponding time period to play back the images through the playback interface.

[0124] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0125] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0126] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the panoramic image monitoring systems and image processing methods described above.

[0127] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above method.

[0128] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus.

[0129] Memory, used to store computer programs;

[0130] The processor, when executing the program stored in the memory, implements the aforementioned panoramic image monitoring system and image processing method.

[0131] The communication bus mentioned in the above-mentioned electronic devices can be a standard bus for interconnecting peripheral components or an extended industrial standard structure bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0132] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0133] The memory may include random access memory or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0134] The processors mentioned above can be general-purpose processors, including central processing units, network processors, etc.; they can also be digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0135] It should also be noted that electronic devices include terminal devices, which can also be called terminals, user equipment, mobile stations, mobile terminals, etc. Terminal devices can be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality terminal devices, augmented reality terminal devices, wireless terminals in industrial control, wireless terminals in autonomous driving, wireless terminals in remote surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on. The embodiments of this application do not limit the specific technologies or device forms used in the terminal devices.

[0136] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0138] Furthermore, it should be noted that if any directional indication (such as up, down, left, right, front, back, etc.) is involved in the embodiments of the present invention, the directional indication is only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0139] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

Claims

1. A panoramic image monitoring system, characterized by, The application relates to a panoramic image acquisition and processing device. The device comprises: an image acquisition module for acquiring visual image data through at least four groups of ultra-wide-angle image acquisition devices of a first device; a data processing module for sequentially performing correction and splicing processing on the acquired image data to obtain panoramic image data, and then performing detection and recognition tracking of a specified target on the processed image, and improving the detection and recognition accuracy of the specified target by using a linear interpolation or curve fitting algorithm; a display module for displaying the processed image data and the detection and recognition tracking result of the specified target; The data processing module further comprises: a preprocessing unit for sequentially performing denoising, brightness adjustment, contrast enhancement and other preprocessing operations on the acquired image data; a distortion correction unit for determining the image distortion part of the image data after the preprocessing operation by using image calibration technology, and performing distortion correction operation according to the device parameters of the ultra-wide-angle image acquisition device; an image splicing unit for identifying the overlapping area and feature points in the image, and splicing multiple images after distortion correction seamlessly by using an image splicing algorithm; a target detection and recognition tracking unit for performing detection and recognition tracking of a specified target on the image data, and improving the detection and recognition accuracy of the specified target after image splicing by using a linear interpolation or curve fitting algorithm; The specific operation process of the image splicing algorithm for seamlessly splicing the image comprises: L1. detecting feature points in the image by using a feature point detection algorithm, describing the feature points as vectors by using a feature descriptor algorithm, and finding matching point pairs by comparing the feature point descriptions; L2. removing the mis-matching points by using a RANSAC algorithm; L3. mapping the feature points in different images to the same coordinate system by using affine transformation or projection transformation, and performing image registration; L4. splicing multiple images to the same canvas, and performing superposition fusion or completion processing; S31. Construct a device body coordinate system with the geometric center of the first device as the origin, the front and back of the moving direction of the first device as the x-axis, the left and right as the y-axis, and the direction perpendicular to the x-axis and the y-axis as the z-axis ; constructing a specified coordinate system and processing the feature point vector data by using the image acquisition device to perform object recognition and tracking operation, and the specific operation process comprises: S33. Let the first The specified feature points in the images acquired by the acquisition devices are Calculate the vector of the feature point relative to the camera coordinate system. By including rotation matrix Translation vector The extrinsic parameter matrix of the acquisition device transforms the feature point vector from the acquisition device coordinate system to the device body coordinate system; S32. feature point vector data processing: extracting feature points by using a feature point detection algorithm such as SIFT, SURF and ORB on the acquired image data; S34. comparing the feature point coordinate changes of the same object at different time in the device body coordinate system to realize object tracking; The device further comprises: a sensor device arranged on the first device for accurately measuring the distance between the first device and surrounding objects, and marking the object position on the panoramic image data; 2. The panoramic video monitoring system of claim 1, wherein, The sensor device collects data for positioning calculation of the actual coordinates of the specified feature points, the midpoint coordinates are calculated by the actual coordinates of the specified feature points in the mapping coordinate system and the recognition coordinates of the mapping coordinate system recognized by the image splicing unit, and the midpoint coordinates are taken as the accurate coordinates of the feature points in the mapping coordinate system, which are used for the detection and recognition tracking processing of the target detection and recognition tracking unit. The operation process of the target detection and recognition tracking unit for detection and recognition tracking of a specified target comprises: Converting the trained deep learning network model into a format suitable for embedded systems and performing forward propagation using model inference libraries such as TensorFlowLite and OpenCV; Adjusting algorithm thresholds and improving prediction accuracy using tracking history, while also verifying algorithm performance based on ground truth data.

3. A panoramic image processing method, based on the panoramic image monitoring system of any one of claims 1 to 2, for embedded data calculation and processing, characterized in that, The method comprises the following steps: S1. For the collected received image data, sequentially perform preprocessing operations such as denoising, brightness and contrast adjustment on each frame of image; S2. After preprocessing, the image data is corrected according to the pre-calibrated acquisition device parameters, and the corrected image data is seamlessly spliced to form complete panoramic image data; S3. Construct a specified coordinate system and process feature point vector data using the image acquisition device to perform object recognition and tracking operations.

4. The panoramic image processing method of claim 3, wherein, The linear interpolation algorithm or curve fitting algorithm is used in the object recognition and tracking operation process in the S3 step to improve the detection and recognition accuracy of the specified target, and the use process of the linear interpolation algorithm includes: Selecting specified feature points of spliced image data , two adjacent pixel points of the feature point and , pixel values of which are and , respectively Calculating pixel values of specified feature points according to a linear interpolation formula , have:​ According to the linear interpolation formula, the pixel value of each feature point in the spliced image is calculated to smooth the image and reduce the discontinuity caused by distortion.

5. The panoramic image processing method of claim 4, wherein, The use process of the curve fitting algorithm in the S34 step includes: Selecting any set of discrete feature points of the stitched image data , , using a quadratic polynomial Fitting the feature points of the image data to optimize the image quality, wherein the specific values of the coefficients a, b, c are determined by the least square method to minimize the sum of squares of errors, have: .

6. A computer device, comprising: The device comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the method according to any one of claims 3 to 5.

Images