Panoramic image monitoring system, image processing method and computer equipment

Abstract

The invention discloses a panoramic image monitoring system, and the system comprises an image collection module which is used for collecting visual image data through at least four groups of ultra-wide-angle image collection devices surrounding a first device; the data processing module is used for sequentially carrying out correction splicing processing on the collected image data to obtain panoramic image data, then carrying out specified target detection and recognition tracking operation on the processed image, and improving the detection and recognition precision of a specified target by adopting a linear interpolation or curve fitting algorithm; and the display module is used for displaying the processed image data and the detection and identification tracking result of the specified target. The method solves the problem that the visual field of the camera may be blocked or the image quality is severely reduced, can effectively improve the stability and reliability of the system, can achieve the precise tracking of the motion state of an object, improves the accuracy and continuity of dynamic target tracking, and improves the user experience. And simultaneous movement or rapid change of a plurality of targets can be conveniently and effectively handled.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a vehicle panoramic image real-time monitoring system, a vehicle panoramic image processing method, a computer device and a storage medium. Background Art

[0002] In modern vehicle safety systems, panoramic image monitoring systems have become one of the key technologies to improve driving safety. For example, the prior art discloses a method, system, and vehicle for generating vehicle panoramic images with patent application number CN201810380394.6, or a method, device, and vehicle for generating panoramic surround view images with patent application number CN202411064436.7.

[0003] A panoramic image monitoring system usually includes cameras installed around the vehicle to capture real-time images of the surrounding environment and process these images through specific algorithms to provide the driver with clear and comprehensive visual information about the vehicle's surroundings. It also uses high-resolution cameras and advanced image processing algorithms to enable the driver to more intuitively understand the conditions around the vehicle, thereby avoiding potential collision risks.

[0004] Conventional panoramic image monitoring systems are mainly used for dynamic target tracking and scene recognition, such as the detection of moving objects such as pedestrians and vehicles. However, in actual applications, when multiple dynamic targets appear at the same time or move quickly, the tracking capabilities of such systems are generally limited, and they are unable to accurately and continuously track each target. Secondly, in special environments such as snow, muddy roads, heavy rain or dense fog, the camera's field of view may be blocked or the image quality may be severely degraded, resulting in reduced system performance or even failure. At the same time, conventional panoramic image monitoring systems are also limited in their ability to identify sharp metal objects or other dangerous obstacles, making it difficult to effectively warn and guide drivers to take avoidance measures. In addition, their ability to prevent the "ghosting" phenomenon (i.e. pedestrians suddenly appearing from blind spots) is also weak.

[0005] To this end, the present application proposes a panoramic image monitoring system, an image processing method and a computer device to solve the above technical problems. Summary of the invention

[0006] The main purpose of the present invention is to provide a panoramic image monitoring system, an image processing method and a computer device. By integrating a 360° camera with no blind spots, a large visual screen and advanced algorithms, it not only enhances the ability to identify various obstacles, but also integrates other sensing devices, which can know the whereabouts of pedestrians in advance and automatically trigger emergency braking measures according to the vehicle speed and distance, thereby effectively preventing traffic accidents caused by emergencies, so as to solve the technical problems raised in the background technology.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] A panoramic image monitoring system, comprising:

[0009] An image acquisition module, configured to acquire visual image data through at least four groups of ultra-wide-angle image acquisition devices surrounding a first device;

[0010] A data processing module, configured to sequentially perform calibration and stitching processing on the acquired image data to obtain panoramic image data, and then perform detection, recognition, and tracking operations on specified targets for the processed images, and use linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of the specified targets;

[0011] A display module, configured to display the processed image data and the detection, recognition, and tracking results of the specified targets.

[0012] Preferably, the data processing module is further internally provided with:

[0013] A preprocessing unit, configured to sequentially perform preprocessing operations including denoising, brightness adjustment, and contrast enhancement on the acquired image data;

[0014] A distortion correction unit, configured to use image calibration technology to determine the image distortion part of the image data after the preprocessing operation, and perform distortion correction operations according to the device parameters of the ultra-wide-angle image acquisition device;

[0015] An image stitching unit, configured to identify the overlapping areas and feature points in the images, and seamlessly stitch multiple distorted-corrected images using an image stitching algorithm;

[0016] A target detection, recognition, and tracking unit, configured to perform detection, recognition, and tracking operations on specified targets for the image data, and use linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of the specified targets after image stitching.

[0017] Preferably, the operation process of the target detection, recognition, and tracking unit for performing detection, recognition, and tracking operations on specified targets includes:

[0018] Converting the trained deep learning network model into a format suitable for an embedded system, and performing forward propagation using model inference libraries including TensorFlowLite and OpenCV;

[0019] Adjusting the algorithm threshold and using the tracking history to improve prediction accuracy, and at the same time verifying the algorithm performance based on ground truth data.

[0020] Preferably, the specific operation process of the image stitching algorithm for seamlessly stitching images includes:

[0021] L1. Detect feature points in the image using a feature point detection algorithm, describe them as vectors using a feature descriptor algorithm, and find matching point pairs by comparing the feature point descriptions;

[0022] L2. Use the RANSAC algorithm to remove mismatched points;

[0023] L3. Map the feature points in different images to the same coordinate system through affine transformation or projective transformation for image registration;

[0024] L4. Stitch multiple images onto the same canvas and perform overlay fusion or completion processing.

[0025] Preferably, it further includes:

[0026] A sensor device provided on the first device for accurately measuring the distance between the first device and surrounding objects and calibrating the object positions on the panoramic image data;

[0027] The sensor device collects data for calculating the actual coordinates of specified feature points. Midpoint coordinate calculation is performed through the actual coordinates of the specified feature points in the mapped coordinate system and the recognition coordinates in the recognition coordinate system of the image stitching unit, and the midpoint coordinates are used as the refined coordinates of the feature points in the mapped coordinate system for specified target detection and recognition tracking processing by the target detection and recognition tracking unit.

[0028] Preferably, a panoramic image processing method performs embedded data calculation and processing based on the panoramic image monitoring system described in any one of the above, including the following steps:

[0029] S1. For the collected image data, perform preprocessing operations on each frame of image in sequence, including denoising, brightness and contrast adjustment;

[0030] S2. After preprocessing, the image data is subjected to distortion correction of the image according to the pre-calibrated acquisition device parameters. After seamless stitching of the distorted-corrected image data, complete panoramic image data is formed;

[0031] S3. Construct a specified coordinate system and use the image acquisition device to process the feature point vector data to perform object recognition and tracking operations.

[0032] Preferably, the specific operation process of performing object recognition and tracking operations in step S3 includes:

[0033] S31. Taking the geometric center of the first device as the origin, with the front and back directions of the movement of the first device as the x-axis, the left and right as the y-axis, and the direction perpendicular to the x-axis and y-axis as the z-axis, construct the device body coordinate system O-xyz;

[0034] S32. Feature point vector data processing: For the collected image data, use feature point detection algorithms such as SIFT, SURF, and ORB to extract feature points;

[0035] S33. Let the specified feature point in the image collected by the i-th acquisition device be P i (x i , y i ), and calculate the vector of this feature point relative to the camera coordinate system Through the external parameter matrix of the acquisition device containing the rotation matrix R i and the translation vector T i , convert the feature point vector from the acquisition device coordinate system to the device body coordinate system. The conversion formula is:

[0036]

[0037] where, (X world , Y world , Z world ) are the coordinates of the feature point in the device body coordinate system.

[0038] S34. Compare the changes in the feature point coordinates of the same object in the device body coordinate system at different times to achieve object tracking. At this time, use the linear interpolation algorithm or the curve fitting algorithm to improve the detection and recognition accuracy of the specified target.

[0039] Preferably, the usage process of the linear interpolation algorithm in step S34 includes:

[0040] Select the specified feature point P(x, y) of the stitched image data. For the two adjacent pixel points P(x1, y1) and P(x2, y2) of this feature point, their pixel values are I1 and I2 respectively;

[0041] Calculate the pixel value I of the specified feature point P according to the linear interpolation formula, and there is:

[0042]

[0043] Calculate the pixel value of each feature point in the stitched image according to the linear interpolation formula to smooth the image and reduce the discontinuity caused by distortion.

[0044] Preferably, the usage process of the curve fitting algorithm in step S34 includes:

[0045] Select any set of discrete feature points (x i , y i ) of the stitched image data, i = 1, 2, 3,..., n, and use the quadratic polynomial y = ax 2Perform fitting processing on the feature points of the image data with \(y = ax^{2}+bx + c\) to optimize the image quality, where the specific values of the coefficients \(a\), \(b\), and \(c\) are determined by the least squares method, which is used to minimize the sum of squared errors to be the minimum, there are:

[0046]

[0047] After obtaining the coefficients \(a\), \(b\), and \(c\) based on the above formula, the quadratic polynomial \(y = ax^{2}+bx + c\) can be used to perform fitting processing on the feature points of the image data 2 +bx + c.

[0048] On the other hand, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to execute the steps of the above method

[0049] On yet another aspect, the present invention also discloses a computer device including a memory and a processor, where the memory stores a computer program, which, when executed by the processor, causes the processor to execute the steps of the above method

[0050] As can be seen from the above technical solutions, the present invention provides a panoramic image monitoring system and an image processing method. Compared with the prior art, the present invention has the following advantages

[0051] 1. By integrating a deep learning network model in the target detection, recognition, and tracking unit and using model inference libraries such as TensorFlow Lite and OpenCV to perform forward propagation, the present invention can automatically adjust the algorithm threshold according to real-time road conditions and improve the prediction accuracy using historical tracking data

[0052] 2. The present invention corrects the recognition results of the feature point coordinates and the algorithm performance based on the ground truth data, ensuring the high accuracy and reliability of the system. Finally, it can effectively prevent the "ghost vehicle" phenomenon and avoid the occurrence of sudden traffic accidents

[0053] 3. Based on the task scheduling mechanism of RTOS, the present invention assigns priorities to different tasks to ensure that time-sensitive tasks are completed on time, and realizes synchronization and communication between tasks through mechanisms such as semaphores and mutexes, thereby ensuring the efficient operation and stability of the entire system

[0054] 4. By adopting linear interpolation or curve fitting algorithms, the present invention improves the detection and recognition accuracy of specified targets. When the image is affected by environmental factors, it can still maintain high image quality and accuracy. It not only solves the problem that the camera's field of view may be blocked or the image quality seriously deteriorates, but also effectively improves the stability and reliability of the system, ensures the normal operation of the image monitoring system under various adverse weather conditions, and enhances driving safety

[0055] 5. By setting up a feature point tracking mechanism in the device body coordinate system, the present invention can compare the changes in the feature point coordinates of the same object at different times, so as to achieve precise tracking of the object's motion state. This not only improves the accuracy and continuity of dynamic target tracking, but also can effectively handle the situation where multiple targets move simultaneously or change rapidly. Finally, it can significantly enhance the system's adaptability to complex traffic environments and ensure driving safety.

[0056] 6. The present invention uses a trained deep learning network model to classify and identify the feature point data mapped to the device body coordinate system, can determine the object category, and take corresponding warning or avoidance measures according to its category. This not only improves the accuracy of object recognition, reduces false alarms and missed detections, but also can adjust the driving strategy in real time according to the recognition result, greatly reducing the risk of traffic accidents caused by failure to detect or correctly identify obstacles in time.

[0057] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Of course, any product implementing the present invention does not necessarily need to achieve all the above-mentioned advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The specification drawings forming a part of this application are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0059] Figure 1 is a schematic diagram of the module framework of the system of the present invention;

[0060] Figure 2 is a schematic diagram of the operation process of image correction, stitching, and target recognition and tracking of the method of the present invention;

[0061] Figure 3 is a schematic diagram of the operation process of image correction and stitching during the use of the system of the present invention;

[0062] Figure 4 is a schematic diagram of the operation process of target tracking, saving, and playback during the use of the system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0063] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0064] In the embodiment, refer in detail to Figures 1 to 4 .

[0065] As Figure 1 shown, a panoramic image monitoring system proposed in an embodiment of the present invention is used for monitoring vehicle panoramic images and target recognition and tracking processing, and includes:

[0066] (1) An image acquisition module, responsible for acquiring image information around the vehicle, for acquiring visual image data through at least four groups of ultra-wide-angle cameras surrounding the vehicle, capable of realizing the acquisition of omnidirectional visual data. Generally, the cameras are installed at the front, rear, left, and right positions of the vehicle. Its main function is to capture and obtain 360° visual information data around the vehicle, so as to achieve the effect of monitoring the vehicle surrounding environment without dead angles, significantly improving the driver's awareness of the surrounding situation, and enhancing driving safety.

[0067] At this time, it should be noted that the ultra-wide-angle cameras used in this application are generally 180-degree or 190-degree ultra-wide-angle fisheye cameras, which have the characteristics of high resolution, high dynamic range, and good low-light performance, ensuring that clear images can be acquired under different lighting conditions. In addition, the wide-angle lens design can cover most areas around the vehicle, and the camera design is small to reduce the impact on the vehicle body wind resistance.

[0068] (2) Sensor devices provided on the vehicle, used to accurately measure the distance between the vehicle and surrounding objects, and calibrate the positions of the objects on the panoramic image data.

[0069] In the actual use process, the sensor devices are arranged around the vehicle. At this time, the system can integrate other types of sensors such as ultrasonic sensors, radars, or lidars (LiDAR). These sensors are used to accurately measure the distance between the vehicle and surrounding objects, and mark the positions of the objects on the panoramic view. Their data is very critical for autonomous driving and assisted parking functions.

[0070] Therefore, the sensor device here collects data for positioning calculation to obtain the actual coordinates of the specified feature points. The midpoint coordinates are calculated by using the actual coordinates of the specified feature points in the mapping coordinate system and the recognized coordinates in the mapping coordinate system identified by the image stitching unit, and the midpoint coordinates are used as the refined coordinates of the feature points in the mapping coordinate system for the specified target detection and recognition tracking processing of the target detection and recognition tracking unit.

[0071] (3) The data processing module is used to perform calibration stitching processing on the collected image data in sequence to obtain panoramic image data, and then perform detection, recognition and tracking operations on the specified targets for the processed images, and use linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of the specified targets. The data processing module is internally provided with:

[0072] (a) The preprocessing unit is used to perform preliminary processing on the collected original images, including operations such as denoising, brightness adjustment, and contrast enhancement, to improve the image quality and provide a better basis for subsequent processing;

[0073] (b) The distortion correction unit. Since the ultra-wide-angle camera will cause image distortion, this unit is used to use image calibration technology to determine the image distortion part of the image data after the preprocessing operation, and perform distortion correction operations according to the device parameters of the ultra-wide-angle camera to make the shape and position of the objects in the image closer to the real situation;

[0074] (c) The image stitching unit is used to identify the overlapping areas and feature points in the images, and use the image stitching algorithm to seamlessly stitch multiple distorted-corrected images to form a complete panoramic image of the vehicle's surroundings;

[0075] The specific operation process of the image stitching algorithm for seamless image stitching includes:

[0076] L1. Use feature point detection algorithms (such as SIFT, SURF, ORB, etc.) to detect feature points in the images, describe them as vectors using feature descriptor algorithms, and find matching point pairs by comparing feature point descriptions through algorithms such as FLANN or brute-force matching;

[0077] L2. Use the RANSAC algorithm to remove mis-matched points;

[0078] L3. Through affine transformation or projective transformation (perspective transformation), map the feature points in different images to the same coordinate system for image registration;

[0079] L4. Stitch multiple images onto the same canvas and perform overlay fusion or completion processing to achieve high-precision and robust image stitching.

[0080] (d) The target detection, recognition and tracking unit is used to detect, recognize and track specified targets such as pedestrians, vehicles, obstacles, road signs, etc. in panoramic image data by using technologies such as deep learning, providing richer environmental information for drivers. In addition, this unit also adopts linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of specified targets after image stitching.

[0081] The operation process of the target detection, recognition and tracking unit for detecting, recognizing and tracking specified targets includes: converting the trained machine learning model (such as a deep learning network) into a format suitable for the embedded system, using a model inference library (such as TensorFlow Lite or OpenCV) to perform forward propagation, then adjusting the algorithm threshold, improving the prediction accuracy by using the tracking history, and verifying the algorithm performance based on the ground truth data. It can automatically adjust the algorithm threshold according to the real-time road conditions and improve the prediction accuracy by using the historical tracking data, ensuring high accuracy while running in real time and reducing false detections and missed detections.

[0082] In summary, during the specific use process of this data processing module, corresponding data processing operations are performed based on a high-performance embedded computing platform, including measures such as the application of a memory pool to avoid fragmentation, reasonable allocation of processor resources to prevent CPU overload, and the use of dynamic power management technology to reduce energy consumption.

[0083] This embedded computing platform is used to coordinate the image acquisition of each camera, process the image data, stitch multiple images into a seamless panoramic image in real time, and at the same time integrate image preprocessing functions such as removing noise and adjusting contrast. It can also process auxiliary data from sensors such as distance information to enhance the accuracy and reliability of the system.

[0084] In addition, during the use process, due to the settings of the preprocessing unit, distortion correction unit, image stitching unit and target detection, recognition and tracking unit, it can effectively remove image noise, adjust brightness and contrast, and perform accurate distortion correction and seamless stitching, thereby improving the image quality. Among them, linear interpolation or curve fitting algorithms are adopted to further improve the detection and recognition accuracy of specified targets. Finally, it can ensure accurate tracking and recognition of dynamic targets in complex environments and reduce the accident risk.

[0085] At this time, in terms of resource management and optimization, this embedded computing platform can accurately control the memory occupation of the program, adopt a memory pool to avoid fragmentation; reasonably allocate processor resources to ensure that key tasks are processed in a timely manner and avoid CPU overload; use dynamic power management technology to reduce energy consumption. At the same time, it also involves cross-compilation, using and maintaining a cross-compilation toolchain to ensure the version compatibility of the compiler, linker and library, adapting to the target platform architecture and instruction set, and an emulator can also be used for preliminary test debugging.

[0086] In addition, regarding the hardware part of the carrier, it can also be supplemented that the image processor is the core hardware of the system of this module, which is responsible for quickly processing a large amount of image data collected by the camera, including complex operations such as distortion correction, image stitching, and target detection. Usually, a high-performance digital signal processor or a dedicated image processor chip is adopted.

[0087] (4) Display module, which is used to display information such as the processed panoramic image data and the detection, recognition, and tracking results of specified targets on the display screen in the vehicle. Usually, it supports 2D or 3D display modes, and the driver can switch different perspectives according to needs, such as bird's-eye view, front view, rear view, side view, etc.

[0088] At this time, it can also be supplemented that the display adopted by this module is installed on the vehicle's center console or dashboard, which is used to display the panoramic image and related information processed by the system. Generally, it is a high-definition and high-contrast liquid crystal display screen. At the same time, this screen is mostly a touch screen, which is convenient for the driver to interact with the system. For example, the driver can select to view the individual view of a specific camera or adjust the system settings. The display screen with high resolution and fast response time can ensure a clear and delay-free view.

[0089] When the display module is interactive, to ensure the real-time response of the system, during the development based on RTOS, its task scheduling, interrupt handling, and time management mechanisms are also utilized to assign priorities to different tasks, ensure that time-sensitive tasks are completed on time, and at the same time, through mechanisms such as semaphores, mutexes, and message queues, the synchronization and communication between tasks are realized, and the data sharing and execution order are coordinated, so as to ensure the efficient operation and stability of the entire system.

[0090] Furthermore, in the software interface part of the display module, the interface layout follows the principle of being clear and easy to understand, and the common function settings have simple operation methods, such as switching views through simple gestures or buttons. The sizes and color contrasts of the interface elements are reasonable, which is convenient for quick recognition and reduces the distraction of the driver's attention. At the same time, the interface can automatically adjust the brightness and color matching according to the ambient light, and provides feedback mechanisms such as touch feedback vibration or audio prompts for successful operations.

[0091] At this time, the interaction logic is close to natural language and intuitive operation, providing voice command recognition or large-size touch buttons, which is convenient for the driver to activate functions. The system has context awareness ability and provides intelligent prompts according to the driving state and environmental conditions, such as automatically activating the parking assist line and providing voice guidance when parking at low speed.

[0092] (5) Recording and playback module: It can record the panoramic images during the vehicle's driving process and store them in the in-vehicle storage device. When an accident occurs or historical images need to be viewed, they can be played back for viewing.

[0093] It can also be supplemented here that the storage device used by this module generally uses a solid-state drive or a large-capacity flash chip, which is used to store the programs, configuration files, and recorded panoramic image data required for the system to run.

[0094] Furthermore, by integrating a 360° panoramic camera, a visualization screen, and advanced algorithms, especially adding an alarm device for identifying sharp metal and non-metal objects, this system can emit a voice alarm when approaching sharp objects to guide the vehicle owner to avoid danger. At the same time, by integrating with infrared and other sensing devices, it can also know the whereabouts of pedestrians in advance and take emergency measures, thus avoiding collision accidents caused by suddenly emerging pedestrians, greatly reducing traffic accidents caused by poor road conditions or unexpected obstacles, and improving the driving safety factor.

[0095] On the other hand, the present invention also discloses a panoramic image processing method, which performs embedded data calculation and processing based on the panoramic image monitoring system proposed in the above embodiment. Referring to Figures 2 to 4 As shown, the overall steps in the actual use process include:

[0096] The first step: After the vehicle starts, the system software first performs initialization operations, including self-checking the hardware devices to check whether devices such as cameras and image processors are working properly; loading the configuration file of the system to obtain parameters of the cameras, algorithm parameters for image stitching, etc.

[0097] The second step: The image acquisition module controls each camera to start working, and acquires image data around the vehicle at a certain frame rate. These image data are transmitted to the image processor in the form of a video stream through a connection line.

[0098] The third step: For the acquired and received image data, after the image processor receives the image data, it performs preprocessing operations on each frame of the image in turn, including denoising, brightness and contrast adjustment, etc., to improve the quality of the image and reduce the influence of factors such as noise and uneven light on subsequent processing.

[0099] The fourth step: After preprocessing, the image data is subjected to distortion correction of the image according to the pre-calibrated parameters of the acquisition device. After the image data after distortion correction from different cameras is seamlessly stitched through a stitching algorithm, a complete panoramic image data of the vehicle periphery is formed.

[0100] The fifth step: Construct a specified coordinate system and use the camera to process the feature point vector data to perform object recognition and tracking operations. At this time, the specific operation process of performing object recognition and tracking operations includes:

[0101] (1) Taking the geometric center of the vehicle as the origin, with the front and back directions of the vehicle's movement as the x-axis, the left and right as the y-axis, and the direction perpendicular to the x-axis and y-axis as the z-axis, a device body coordinate system O-xyz is constructed;

[0102] (2) Feature point vector data processing: For the collected image data, use feature point detection algorithms including SIFT, SURF, and ORB to extract feature points;

[0103] (3) Let the specified feature point in the image collected by the i-th acquisition device be P i (x i , y i ). Calculate the vector of this feature point relative to the camera coordinate system Through the external parameter matrix of the acquisition device containing the rotation matrix R i and the translation vector T i , convert the feature point vector from the acquisition device coordinate system to the device body coordinate system. The conversion formula is:

[0104]

[0105] Among them, (X world , Y world , Z world ) are the coordinates of the feature point in the device body coordinate system.

[0106] (4) Compare the changes in the feature point coordinates of the same object in the device body coordinate system at different times to achieve object tracking. For example, let the coordinates of the feature point on an object at time t1 in the device body coordinate system be (X1, Y1, Z1), and the coordinates of this feature point become (X2, Y2, Z2) at time t2. Then the displacements of the object in the x, y, and z directions are ΔX = X2 - X1, ΔY = Y2 - Y1, ΔZ = Z2 - Z1 respectively. According to these displacement information, the motion state of the object can be analyzed.

[0107] At this time, by setting a feature point tracking mechanism in the device body coordinate system, it is possible to compare the changes in the feature point coordinates of the same object at different times, thereby achieving precise tracking of the object's motion state. This not only improves the accuracy and continuity of dynamic target tracking but also effectively deals with the situation where multiple targets move simultaneously or change rapidly. Finally, it can significantly enhance the system's adaptability to complex traffic environments and ensure driving safety.

[0108] In addition, for object recognition, a trained machine learning model (such as a deep learning network) can be used to classify and recognize the feature point data mapped to the device body coordinate system to determine the category of the object (such as pedestrians, vehicles, sharp objects, etc.).

[0109] Furthermore, in the process of detecting and recognizing a specified target based on feature points, a linear interpolation algorithm or a curve fitting algorithm can also be used to improve the detection and recognition accuracy of the specified target. The usage process of the linear interpolation algorithm includes:

[0110] Select a specified feature point P(x, y) of the stitched image data. For two adjacent pixel points P(x1, y1) and P(x2, y2) of this feature point, their pixel values are I1 and I2 respectively;

[0111] Calculate the pixel value I of the specified feature point P according to the linear interpolation formula, as follows:

[0112]

[0113] Calculate the pixel value of each feature point in the stitched image according to the linear interpolation formula, which is used to smooth the image and reduce the discontinuity caused by distortion;

[0114] The usage process of the curve fitting algorithm includes:

[0115] Select any set of discrete feature points (x i , y i ) of the stitched image data, where i = 1, 2, 3,..., n. Use the quadratic polynomial y = ax 2 + bx + c to fit the feature points of the image data to optimize the image quality. The specific values of the coefficients a, b, and c are determined by the least squares method, which is used to minimize the sum of squared errors to be the minimum, as follows:

[0116]

[0117] After obtaining the coefficients a, b, and c based on the above formula, the quadratic polynomial y = ax 2 + bx + c can be used to fit the feature points of the image data, thereby adjusting the pixel values, optimizing the image quality, improving the object recognition accuracy and the accuracy of sharp object determination. In addition, in the specific implementation process, compared with the linear interpolation, the curve fitting algorithm can better adapt to complex image distortion situations and more effectively reduce the distortion correction error.

[0118] In summary, using a linear interpolation or curve fitting algorithm to improve the detection and recognition accuracy of a specified target can still maintain high image quality and accuracy when the image is affected by environmental factors. It not only solves the problem that the camera's field of view may be blocked or the image quality seriously deteriorates, but also effectively improves the stability and reliability of the system, ensures that the image monitoring system works normally under various bad weather conditions, and enhances driving safety.

[0119] In addition, the coordinate recognition results of feature points and the algorithm performance can be corrected according to the ground truth data, ensuring the high accuracy and reliability of the system. Finally, the "ghost vehicle" phenomenon can be effectively prevented, and sudden traffic accidents can be avoided.

[0120] Step 6: Use a deep learning algorithm to perform object detection and recognition on the stitched panoramic image, identify objects such as pedestrians, vehicles, obstacles, and road signs in the image, and locate and track them. Integrate the detected object information with the panoramic image to provide more intuitive environmental information for the driver.

[0121] The above method of classifying and recognizing the feature point data mapped to the device body coordinate system using the trained deep learning network model can determine the object category and take corresponding warning or avoidance measures according to its category. This can not only improve the accuracy of object recognition, reduce false alarms and missed detections, but also adjust the driving strategy in real time according to the recognition results, significantly reducing the risk of traffic accidents caused by failure to detect or correctly identify obstacles in a timely manner.

[0122] Step 7: Send the processed panoramic image and object detection results to the display module for display on the in-vehicle display. The driver can switch different display perspectives through interaction devices such as the vehicle's control buttons or touch screen to view the detailed situation around the vehicle. At the same time, the system can also automatically switch to the corresponding perspective according to the driving state of the vehicle, such as reversing or turning.

[0123] Step 8: The recording and playback module records the panoramic video in real time according to the set rules during the vehicle's driving process and stores the image data in the in-vehicle storage device. When the driver needs to view the historical video, they can select the corresponding time period for video playback by operating the playback interface.

[0124] On the other hand, the present invention also discloses a computer-readable storage medium storing a computer program, which when executed by a processor, causes the processor to execute the steps of the above method.

[0125] On yet another aspect, the present invention also discloses a computer device including a memory and a processor, the memory storing a computer program, which when executed by the processor, causes the processor to execute the steps of the above method.

[0126] In another embodiment provided by the present application, a computer program product containing instructions is also provided, which when running on a computer, causes the computer to execute any of the panoramic image monitoring systems and image processing methods in the above embodiments.

[0127] It is understandable that the system provided by the embodiments of the present invention corresponds to the method provided by the embodiments of the present invention. For the explanations, examples, and beneficial effects of related content, reference can be made to the corresponding parts in the above method.

[0128] An embodiment of the present application further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus.

[0129] The memory is used to store a computer program.

[0130] When the processor is used to execute the program stored on the memory, it implements the above panoramic image monitoring system and image processing method.

[0131] The communication bus mentioned in the above electronic device can be a peripheral component interconnect standard bus or an extended industry standard architecture bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0132] The communication interface is used for communication between the above electronic device and other devices.

[0133] The memory can include a random access memory and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located far from the aforementioned processor.

[0134] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0135] It should also be noted that the electronic device further includes a terminal device, which can also be called a terminal, a user equipment, a mobile station, a mobile terminal, etc. The terminal device can be a mobile phone, a smart TV, a wearable device, a tablet computer, a computer with wireless transceiver function, a virtual reality terminal device, an augmented reality terminal device, a wireless terminal in industrial control, a wireless terminal in unmanned driving, a wireless terminal in remote surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the terminal device.

[0136] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive), etc.

[0137] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0138] In addition, it should be noted that if there are directional indications (such as up, down, left, right, front, back...) involved in the embodiments of the present invention, the directional indications are only used to explain the relative positional relationship and movement conditions between components in a specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0139] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the meaning of "and / or" appearing throughout the text includes three parallel scenarios. Taking "A and / or B" as an example, it includes scenario A, scenario B, or the scenario where A and B are satisfied simultaneously. In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on the fact that those skilled in the art can implement them. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

Claims

1. A panoramic image monitoring system, characterized in that: include: An image acquisition module, used to acquire visual image data through at least four sets of ultra-wide-angle image acquisition devices surrounding the first device; The data processing module is used to sequentially correct and stitch the collected image data to obtain panoramic image data, and then perform detection, identification and tracking operations on the processed images for the designated targets, and use linear interpolation or curve fitting algorithms to improve the detection and identification accuracy of the designated targets; The display module is used to display the processed image data and the detection, recognition and tracking results of the specified target.

2. The panoramic image monitoring system according to claim 1, characterized in that: The data processing module is also provided with: A preprocessing unit, used to perform preprocessing operations including denoising, brightness adjustment and contrast enhancement on the collected image data in sequence; A distortion correction unit, used to determine the image distortion part of the image data after the preprocessing operation by using the image calibration technology, and perform the distortion correction operation according to the device parameters of the ultra-wide-angle image acquisition device; An image stitching unit, used to identify overlapping areas and feature points in images, and seamlessly stitch multiple distortion-corrected images using an image stitching algorithm; The target detection, recognition and tracking unit is used to detect, recognize and track the specified target on the image data, and use linear interpolation or curve fitting algorithms to improve the detection and recognition accuracy of the specified target after image stitching.

3. The panoramic image monitoring system according to claim 2, characterized in that: The target detection, identification and tracking unit performs detection, identification and tracking of a specified target, including: Convert trained deep learning network models into a format suitable for embedded systems and perform forward propagation using model inference libraries including TensorFlowLite and OpenCV; Adjust algorithm thresholds and leverage tracking history to improve prediction accuracy while also validating algorithm performance against ground truth data.

4. The panoramic image monitoring system according to claim 2, characterized in that: The specific operation process of the image stitching algorithm for seamless image stitching includes: L1. Use feature point detection algorithm to detect feature points in the image, and describe them as vectors using feature descriptor algorithm, and find matching point pairs by comparing feature point descriptions; L2. Use RANSAC algorithm to remove mismatched points; L3. Through affine transformation or projection transformation, feature points in different images are mapped to the same coordinate system for image registration; L4. Stitch multiple images onto the same canvas and perform overlay, fusion or completion processing.

5. The panoramic image monitoring system according to claim 4, characterized in that: Also includes: A sensor device disposed on the first device, used to accurately measure the distance between the first device and surrounding objects and calibrate the position of the object on the panoramic image data; The sensor device collects data for locating and calculating the actual coordinates of the specified feature point, calculates the midpoint coordinates by the actual coordinates of the specified feature point in the mapping coordinate system and the identification coordinates in the mapping coordinate system identified by the image stitching unit, and uses the midpoint coordinates as the precise coordinates of the feature point in the mapping coordinate system for the specified target detection and identification tracking processing of the target detection, identification and tracking unit.

6. A panoramic image processing method, based on the panoramic image monitoring system according to any one of claims 1 to 5, for embedded data calculation and processing, characterized in that: The following steps are involved: S1. For the image data received, pre-processing operations including denoising, brightness and contrast adjustment are performed on each frame of the image in sequence; S2. After the preprocessing, the image data is subjected to distortion correction according to the pre-calibrated acquisition equipment parameters, and the image data after the distortion correction is seamlessly spliced ​​to form a complete panoramic image data; S3. Construct a specified coordinate system and use an image acquisition device to process feature point vector data to perform object recognition and tracking operations.

7. The panoramic image processing method according to claim 6, wherein: The specific operation process of performing the object recognition and tracking operation in step S3 includes: S31. With the geometric center of the first device as the origin, the front and back moving direction of the first device as the x-axis, the left and right as the y-axis, and the direction perpendicular to the x-axis and the y-axis as the z-axis, construct the device body coordinate system O-xyz; S32. Feature point vector data processing: For the collected image data, feature point detection algorithms including SIFT, SURF, and ORB are used to extract feature points; S33. Let the specified feature point in the image collected by the i-th acquisition device be P i (x i ,y i ), calculate the vector of the feature point relative to the camera coordinate system By including the rotation matrix R i and the translation vector T i The acquisition device external parameter matrix transforms the feature point vector from the acquisition device coordinate system to the device body coordinate system; S34. Compare the coordinate changes of feature points of the same object in the device body coordinate system at different times to achieve object tracking. At this time, a linear interpolation algorithm or a curve fitting algorithm is used to improve the detection and recognition accuracy of the specified target.

8. The panoramic image processing method according to claim 7, wherein: The use process of the linear interpolation algorithm in step S34 includes: Select a specified feature point P(x, y) of the stitched image data, and for the two adjacent pixel points P(x1, y1) and P(x2, y2) of the feature point, their pixel values ​​are I1 and I2 respectively; According to the linear interpolation formula, the pixel value I of the specified feature point P is calculated as follows: The pixel value of each feature point in the stitched image is calculated according to the linear interpolation formula to smooth the image and reduce the discontinuity caused by distortion.

9. The panoramic image processing method according to claim 7, wherein: The use process of the curve fitting algorithm in step S34 includes: Select any set of discrete feature points (x i ,y i ), i = 1, 2, 3, ..., n, using the quadratic polynomial y = ax 2 +bx+c performs fitting processing on the feature points of the image data to optimize the image quality. The specific values ​​of coefficients a, b, and c are determined by the least squares method to minimize the sum of squared errors:

10. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 6 to 9.

Images