Image processing method and device based on look-around system, vehicle and storage medium

Abstract

The embodiment of the invention provides an image processing method and device based on a look-around system, a vehicle and a storage medium. The method comprises the following steps: recording an image with earliest acquisition time as a first real-time image and an image with acquisition time later than that of the first real-time image as a second real-time image according to a current image set acquired by a surround view system; acquiring an acquisition time difference between the first real-time image and each second real-time image, and performing image frame loss detection on the current image set according to all the acquisition time differences; after it is determined that no image frame loss exists in the current image set, obtaining a to-be-spliced image set through time synchronization according to the collection timestamp of the first real-time image; and according to the to-be-stitched image set and the first real-time image, performing all-round image stitching to obtain an all-round stitched image. According to the method and the device, a plurality of image data sources with relatively high time consistency are acquired, so that splicing errors caused by acquisition time are reduced, and a driver can conveniently master the environment around the vehicle.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, vehicle and storage medium based on a surround view system. Background Art

[0002] During the driving process, the surrounding environment, the vehicle's own status and obstacles often change constantly. As a vehicle assisted driving technology, the surround view system can collect image data of the vehicle's surrounding environment through multiple cameras and generate a 360-degree panoramic surround image through an image stitching algorithm. The surround view image can be displayed in real time on the vehicle's central control screen, so that the driver can intuitively view the situation around the vehicle, including the location, type and movement trajectory of obstacles.

[0003] In the prior art, a surround view system usually includes cameras installed in multiple locations of the vehicle (such as the front, rear, and left and right sides). These cameras collect image data of the vehicle's surroundings at a preset sampling frequency. The image processing unit processes and splices images from different cameras through an image stitching algorithm to obtain a surround view image. Existing surround view systems usually assume that the image acquisition time of each camera is completely synchronized. However, in actual applications, due to factors such as hardware differences between cameras, data transmission delays, and external environmental interference, there is often a certain deviation in the image acquisition time of different cameras.

[0004] When the environment around the vehicle changes, if images with large time deviations are stitched together, errors such as ghosting and misalignment may occur in the surround view image, making it impossible for the driver to accurately judge the actual situation around the vehicle. This will not only affect the driver's judgment, but may also lead to false warnings or missed warnings, thus affecting driving safety. Based on this, an image processing method is urgently needed. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, device, vehicle and storage medium based on a surround view system, which are used to reduce the error of a surround view stitched image and make the surround view stitched image have higher real-time performance and continuity.

[0006] In a first aspect, an embodiment of the present application provides an image processing method based on a surround view system, comprising:

[0007] Receiving a current image set acquired by the surround view system, and based on the current image set, recording an image with the earliest acquisition time as a first real-time image, and recording an image acquired later than the first real-time image as a second real-time image;

[0008] Based on the current image set, acquiring an acquisition time difference between the first real-time image and each of the second real-time images according to acquisition timestamps of the first real-time image and all the second real-time images, and performing image frame loss detection on the current image set according to all the acquisition time differences;

[0009] After determining that there is no image frame loss in the current image set, acquiring a set of images to be stitched corresponding to the acquisition timestamp according to the acquisition timestamp of the first real-time image through time synchronization; wherein the set of images to be stitched includes the second real-time image and / or historical images in the previous image set;

[0010] Perform surround image stitching on the second real-time image and / or the historical image in the set of images to be stitched, and the first real-time image, to obtain a surround stitching image, and complete image processing.

[0011] In a possible implementation, acquiring a set of images to be stitched corresponding to the acquisition timestamp of the first real-time image through time synchronization includes:

[0012] According to the acquisition time difference between the first real-time image and the second real-time image, determining whether the acquisition time difference is less than or equal to a first time threshold;

[0013] If the acquisition time difference is less than or equal to the first time threshold, the second real-time image is stored in the set of images to be stitched, so as to obtain the set of images to be stitched.

[0014] In a possible implementation manner, acquiring the set of images to be stitched corresponding to the acquisition timestamp of the first real-time image through time synchronization further includes:

[0015] If the acquisition time difference is greater than the first time threshold, acquiring a data source tag carried by the second real-time image according to the second real-time image corresponding to the acquisition time difference;

[0016] Acquire a previous image set acquired by the surround view system, and acquire a historical image corresponding to the data source label from the previous image set;

[0017] According to the acquisition timestamp of the first real-time image and the acquisition timestamp of the historical image, after determining that the acquisition time difference between the first real-time image and the historical image is less than or equal to a first time threshold, the historical image is stored in the set of images to be stitched to obtain the set of images to be stitched.

[0018] In a possible implementation manner, performing image frame loss detection on the current image set according to all the acquisition time differences includes:

[0019] Determine whether the acquisition time difference is less than or equal to a second time threshold;

[0020] If all the acquisition time differences are less than or equal to the second time threshold, it is determined that there is no image frame loss in the current image set; or,

[0021] If any of the acquisition time differences is greater than the second time threshold, it is determined that image frame loss exists in the current image set, and an alarm message is sent to the vehicle central control terminal so that the vehicle central control terminal can visualize the alarm message.

[0022] In a possible implementation, performing surround image stitching on the second real-time image and / or the historical image in the set of images to be stitched, and the first real-time image, to obtain a surround stitched image includes:

[0023] Recording the second real-time image and / or the historical image in the set of images to be stitched, and the first real-time image, as images to be processed;

[0024] According to all the images to be processed, an image segmentation algorithm is used to identify and obtain feature points of each image to be processed;

[0025] Acquire an image transformation matrix of each image to be processed according to the calibration parameters of the surround view system and the feature points of all the images to be processed;

[0026] According to the image transformation matrix of all the images to be processed, a corrected image corresponding to each image to be processed is obtained by adjusting parameters;

[0027] According to the correspondence between the feature points, the corrected images are stitched together to obtain a surround stitching image, and the surround stitching image is sent to the vehicle central control terminal so that the central control terminal can visualize the surround stitching image.

[0028] In a possible implementation, after receiving the current image set acquired by the surround view system, the method further includes:

[0029] Acquire the number of image acquisition devices in the surround view system, and determine whether the preset image acquisition devices in the surround view system work normally according to the number of images in the current image set;

[0030] If the number of images is less than the number of image acquisition devices, reading and analyzing the data source tags carried by all images in the current image set to screen out faulty image acquisition devices that are not working properly;

[0031] After determining that the abnormal time of the fault image acquisition device meets the third time threshold, the surround view stitching image carrying the fault warning information is sent to the vehicle central control terminal, so that the vehicle central control terminal can visualize the surround view stitching image and the fault warning information.

[0032] In a possible implementation, the current image set acquired by the surround view system is received through a preset wireless home digital interface.

[0033] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0034] An information acquisition module, configured to receive a current image set acquired by the surround view system, and based on the current image set, record an image with the earliest acquisition time as a first real-time image, and record an image with a later acquisition time than the first real-time image as a second real-time image;

[0035] a frame loss detection module, configured to obtain, based on the current image set and according to acquisition timestamps of the first real-time image and all the second real-time images, an acquisition time difference between the first real-time image and each of the second real-time images, and perform image frame loss detection on the current image set according to all the acquisition time differences;

[0036] A time synchronization module, configured to, after determining that there is no image frame loss in the current image set, obtain, according to the acquisition timestamp of the first real-time image, a set of images to be stitched corresponding to the acquisition timestamp through time synchronization; wherein the set of images to be stitched includes the second real-time image and / or historical images in the previous image set;

[0037] The image stitching module is used to stitch the second real-time image and / or the historical image in the set of images to be stitched, and the first real-time image, to obtain a stitched image and complete image processing.

[0038] In a third aspect, an embodiment of the present application provides a vehicle, including: a memory, a processor;

[0039] The memory stores computer-executable instructions;

[0040] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0043] The image processing method, device, vehicle and storage medium based on the surround view system provided by the embodiment of the present application. The present application receives the current image set acquired by the surround view system, and determines the first real-time image and the second real-time image according to the acquisition timestamp of the image in the current image set, thereby realizing the time sorting and classification processing of the image data. The temporal sequence of the image data is ensured, and the time foundation for the subsequent image splicing and processing is laid. By identifying the image with the earliest acquisition time as the first real-time image, and classifying other images as the second real-time image, the problems of processing confusion and inefficiency caused by disordered image data are avoided. After obtaining the acquisition time difference between the first real-time image and each second real-time image, the current image set is subjected to image frame loss detection, and the integrity of the image data is verified, ensuring that the data required for subsequent image splicing is continuous and complete. Frame loss detection reduces the surround view image splicing errors caused by data loss, thereby improving the accuracy and consistency of the surround view image. Through time synchronization, a set of images to be spliced ​​is obtained, and time synchronization not only ensures that each image data in the set of images to be spliced ​​has a high consistency in time, but also reduces the splicing error caused by time deviation. The acquisition process of the image set to be stitched, including the second real-time image in the current image set and the historical image in the previous image set, ensures the diversity and integrity of the data source. This multi-data source integration method can flexibly respond to different image acquisition situations and ensure that high-quality stitched images can be obtained in various driving environments. The second real-time image and / or historical image in the image set to be stitched, as well as the first real-time image, are stitched together to improve the practicality and accuracy of the vehicle's surround view system. In the image stitching process, the stitching of surround view images is achieved based on multiple image data sources with high temporal consistency, allowing the driver to fully understand the real-time environmental conditions around the vehicle and improve the driving safety of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] Figure 1A flowchart of an image processing method based on a surround view system provided in an embodiment of the present application;

[0046] Figure 2 A schematic flow chart of a method for obtaining a set of images to be stitched provided in an embodiment of the present application;

[0047] Figure 3 A schematic flow chart of another method for obtaining a set of images to be stitched provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a method flow for performing image frame loss detection provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a method flow for obtaining a surround stitching image provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of a method flow chart for detecting whether a surround view system is operating normally provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of the image processing device provided by this application;

[0052] Figure 8 A schematic diagram of the structure of the vehicle provided for this application;

[0053] Fig. 9 This is a schematic diagram of the structure of a vehicle based on wireless transmission according to an embodiment of the present application.

[0054] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0056] The surround view system can collect image data of the vehicle's surroundings through multiple cameras, and generate a 360-degree panoramic surround view image through an image stitching algorithm. When the image acquisition time of different cameras is not completely synchronized, the image stitching algorithm may produce stitching errors due to the time difference, resulting in a decrease in the real-time and stability of the surround view image, thereby affecting driving safety. At the same time, the prior art generally lacks a frame loss detection mechanism for image data. If data is lost during the image acquisition and transmission process, incomplete image data may be directly used for stitching, resulting in faults, blurring or freezing of the surround view image, thereby affecting the driver's judgment and operation, making it difficult for the driver to make decisions quickly and accurately.

[0057] Based on the above problems and needs, the inventive concept of the present application is to manage and filter multiple images acquired at the same time (or adjacent time) in time sequence, and then unify these images in time and space to achieve high-precision surround stitching. First, the current image set acquired by the surround system is received, and the image with the earliest acquisition time is identified as the first real-time image, and the other relatively late images are regarded as the second real-time images. Subsequently, by summarizing and analyzing the acquisition time difference between the first real-time image and each second real-time image, it is possible to quickly detect whether there is an acquisition anomaly such as frame loss; if it is confirmed that there is no frame loss problem in the current image set, then according to the acquisition timestamp of the first real-time image, the image set to be stitched is obtained by time synchronization, which includes the second real-time image that may be shortly different, and some historical images may be retrieved from the previous image set for time alignment. Finally, these images (including the first real-time image, the second real-time image and / or the historical image) are stitched for surround view images to obtain surround view stitching images that maintain high coherence and consistency in both time sequence and space, and complete the image processing of this batch. Through such a processing flow, the present application solves the problems of timing misalignment or image loss that may occur in the image acquisition, synchronization and stitching links of traditional surround view methods, ensuring that accurate and stable panoramic views can still be generated in a multi-source, multi-time data environment, providing reliable support for subsequent vehicle safety assistance and environmental perception.

[0058] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0059] Figure 1 The process diagram of the image processing method based on the surround view system provided in the embodiment of the present application is as follows: Figure 1 As shown, the method includes:

[0060] S11, receiving a current image set acquired by a surround view system, and based on the current image set, recording an image acquired earliest as a first real-time image, and recording an image acquired later than the first real-time image as a second real-time image.

[0061] During the driving process of the vehicle, the surround view system collects image information of the vehicle's surrounding environment in real time from different angles and different positions. These images usually have different acquisition times. Therefore, in order to ensure the time synchronization and data integrity of subsequent image splicing, it is necessary to archive and manage these images in time order, that is, to divide the time sequence relationship of the images. Specifically, first, while recording each image information, an acquisition timestamp is assigned to identify the specific moment of image acquisition. Subsequently, by comparing the acquisition timestamps of different images, the first image collected in the current image set is found and recorded as the first real-time image. Since this image is collected the earliest, it is regarded as the benchmark for the image processing of this batch. In contrast, the remaining images collected later are recorded as the second real-time image for time sequence pairing or synchronization with the first real-time image. Through the implementation of this step, the time consistency and integrity of all image data can be ensured before image splicing, which provides basic support for subsequent image frame loss detection, time synchronization and surround view image splicing. In addition, the use of a timestamp-based image recording method can effectively avoid splicing errors caused by time deviations between different image acquisition devices, thereby improving the accuracy and real-time performance of surround view splicing images.

[0062] S12, based on the current image set, according to the acquisition timestamps of the first real-time image and all the second real-time images, obtain the acquisition time difference between the first real-time image and each second real-time image, and perform image frame loss detection on the current image set according to all the acquisition time differences.

[0063] In this embodiment, the current image set has recorded the acquisition timestamps of the first real-time image and all the second real-time images. In order to ensure the integrity of image acquisition, it is necessary to perform time difference analysis on these images. By comparing the acquisition timestamps of the first real-time image and each second real-time image, several acquisition time differences are calculated, each of which corresponds to the acquisition time difference between the first real-time image and a second real-time image. Subsequently, it is determined whether these time differences conform to the predefined time sampling law, for example, whether they are within a reasonable time resolution range or below a specific threshold. If all acquisition time differences are within an acceptable range, it is determined that the current image set is coherent on the time axis, and there are no anomalies such as missing frames, frame skipping, or excessive image delay. If any time difference is detected to deviate from the set range (for example, exceeding the maximum allowed time interval), it may be determined that there is image frame loss in the current image set, and the corresponding exception handling or alarm mechanism is triggered. Through this process of time difference calculation and inspection, the continuity of image acquisition can be effectively monitored, thereby ensuring the integrity and reliability of the data source in the subsequent image processing or splicing stage, and avoiding surround image splicing errors caused by image frame loss.

[0064] S13, after determining that there is no image frame loss in the current image set, according to the acquisition timestamp of the first real-time image, through time synchronization, obtain the image set to be stitched corresponding to the acquisition timestamp; wherein the image set to be stitched includes the second real-time image and / or historical images in the previous image set.

[0065] In this embodiment, in order to ensure the temporal continuity and spatial consistency of the image, after completing the frame loss detection for the current image set and confirming that there is no image frame loss, it is necessary to obtain the image set to be spliced ​​corresponding to the acquisition timestamp of the first real-time image from the current image set and / or the previous image set through the time synchronization mechanism based on the acquisition timestamp of the first real-time image. The core purpose of this process is to provide a set of time-synchronized high-quality image data for surround image splicing, thereby improving the integrity and accuracy of the spliced ​​image. First, read the acquisition timestamp of the first real-time image and use it as the reference time point for time synchronization. Through the acquisition timestamp of the first real-time image, it is possible to determine which second real-time images in the current image set have an acquisition time difference with the first real-time image within a reasonable range, that is, which images can be regarded as time-synchronized image data. If some second real-time images in the current image set fail to meet the time synchronization requirements, it is necessary to further search for qualified historical images from the previous image set of the surround system. The previous image set refers to the image data acquired and stored by the surround system in the previous image acquisition cycle. These historical images may be closer to the acquisition timestamp of the first real-time image in time, and thus more suitable as part of the spliced ​​image. According to the timestamp comparison, historical images that meet the time synchronization mechanism are screened out, and these images are added to the set of images to be stitched. In summary, through time synchronization, the set of images to be stitched that match the acquisition timestamp is screened and obtained. This process not only focuses on the second real-time image in the current image set, but also may retrieve historical images from the previous image set to achieve temporal alignment. Provide a reliable data foundation for subsequent image stitching and scene restoration. The set of images to be stitched therefore covers image data from multiple times and sources. Under a unified time reference, the continuity and information integrity of the surround view image are guaranteed, which helps to finally stitch together a clear and coherent view of the scene around the vehicle.

[0066] S14, performing surround image stitching on the second real-time image and / or the historical image in the image set to be stitched, and the first real-time image, to obtain a surround stitching image, and complete image processing.

[0067] In this embodiment, after time synchronization is completed, the second real-time image and / or historical image in the set of images to be stitched, as well as the first real-time image, will be centrally included in the stitching process for image stitching. Exemplarily, based on the overlapping areas between these images, an image fusion algorithm is used to eliminate seams and differences, thereby generating a coherent and consistent surround stitching image. In this process, each image is matched with the first real-time image to ensure that the stitched view fully presents the spatial layout of the environment around the vehicle. In this way, the surround stitching image can be acquired and the image processing can be completed, and the multi-perspective real-time images around the vehicle can be integrated into an intuitive panoramic view, which is convenient for subsequent monitoring and display.

[0068] This application records the image with the earliest acquisition time as the first real-time image and regards the subsequent images as the second real-time image. After receiving the current image set, the acquisition order of each image can be determined, thereby realizing rapid distinction of image timing. By calculating the acquisition time difference between the first real-time image and each second real-time image, and performing image frame loss detection, a set of closely connected image data is obtained on the time axis. After confirming that there is no image frame loss in the current image set, the second real-time image corresponding to the acquisition timestamp of the first real-time image and / or the historical image in the previous image set can be included in the image set to be spliced ​​through time synchronization, so as to realize effective archiving of images from different times and sources. Finally, the image set to be spliced ​​is spliced ​​with the first real-time image for surround view image splicing to form an overall coherent surround view spliced ​​image. Through the above operation, while maintaining high-efficiency processing, the accuracy and real-time performance of the surround view spliced ​​image are improved, providing a more reliable visual display for vehicle surrounding environment perception and driving safety assistance.

[0069] In one embodiment, Figure 2 The following is a flow chart of a method for obtaining a set of images to be stitched provided in an embodiment of the present application. It is an explanation of an implementation method for obtaining a set of images to be stitched in the above step S13. Based on the above embodiment, Figure 2 As shown, including:

[0070] S21, judging whether the acquisition time difference between the first real-time image and the second real-time image is less than or equal to a first time threshold according to the acquisition time difference between the first real-time image and the second real-time image;

[0071] S22: If the acquisition time difference is less than or equal to the first time threshold, the second real-time image is stored in the set of images to be stitched, so as to obtain the set of images to be stitched.

[0072] In this embodiment, the acquisition time difference between the first real-time image and each second real-time image is calculated one by one according to the acquisition timestamp of the first real-time image and the acquisition timestamp of each second real-time image. If the time difference is less than or equal to the first time threshold, it indicates that the two images are close enough in time and can be regarded as images acquired at approximately synchronous moments. Here, the first time threshold is usually a maximum time allowance set according to the sampling frequency or the system's real-time requirements to limit the timing deviation range of image acquisition. For example, in a vehicle's surround view system, the image acquisition frequency is usually 10 to 30 frames per second, so the first time threshold can be set to 30 milliseconds to 100 milliseconds to ensure that the time difference is within a reasonable range.

[0073] Determine in turn whether the acquisition time difference of each second real-time image is less than or equal to the first time threshold. And store the second real-time image with a time difference within the threshold range into the set of images to be stitched for subsequent image stitching. The set of images to be stitched is a group of images used to collect and store images that will be stitched in subsequent operations, so that all images included in the set are sufficiently matched in time sequence, thereby ensuring higher stitching accuracy and coherence in subsequent time synchronization and transformation processing. If the acquisition time difference corresponding to a second real-time image exceeds the first time threshold, it means that the image may have acquisition delays or frame loss due to network delays, equipment failures, etc. In this case, the second real-time image will not be stored in the set of images to be stitched. It can be seen that by analyzing the time difference between the two images and comparing them with the first time threshold, the consistency screening of the images in the time dimension is achieved, and it is ensured that only the second real-time image that is close to the first real-time image in terms of acquisition time will enter the set of images to be stitched, thereby reducing the scene dislocation or target position offset caused by the large time span during subsequent stitching, improving the integrity and accuracy of the stitched image, and ensuring that the surround view image seen by the driver can truly and accurately reflect the environmental conditions around the vehicle.

[0074] Next, in another embodiment, Figure 3 The following is a flow chart of another method for obtaining a set of images to be stitched provided in an embodiment of the present application. It is an explanation of another implementation method for obtaining a set of images to be stitched in the above step S13. Figure 3 As shown, including:

[0075] S31, if the acquisition time difference is greater than the first time threshold, acquiring a data source tag carried by the second real-time image according to the second real-time image corresponding to the acquisition time difference;

[0076] S32, obtaining a previous image set acquired by the surround view system, and obtaining a historical image corresponding to the data source label from the previous image set;

[0077] S33, according to the acquisition timestamp of the first real-time image and the acquisition timestamp of the historical image, after determining that the acquisition time difference between the first real-time image and the historical image is less than or equal to the first time threshold, storing the historical image in the set of images to be stitched to obtain the set of images to be stitched.

[0078] In this embodiment, when it is detected that the acquisition time difference between the first real-time image and the second real-time image is greater than the first time threshold, it means that the acquisition time of the two images is relatively scattered and is not suitable for image stitching as the same batch. To this end, it is necessary to obtain the data source tag carried by the image according to the second real-time image corresponding to the time difference. Here, the data source tag refers to an identifier used to identify the specific source of the image or the camera channel information, which is automatically attached to the image metadata during image acquisition for subsequent positioning and retrieval of information associated with this image. Exemplarily, the data source tag usually contains information such as device number, camera position, and acquisition direction. After obtaining the data source tag, the historical image corresponding to the data source tag is retrieved from the previous image set previously collected and saved by the surround view system. The so-called previous image set refers to a set of surround view images that have been stored and not yet eliminated or covered in the previous batch of image acquisition. These images also carry metadata information such as data source tags to achieve data association in image processing of different batches. The time difference is calculated again by comparing the acquisition timestamps of the first real-time image and the retrieved historical image. When it is found that the acquisition time difference between the two is less than or equal to the first time threshold, it means that the historical image is sufficiently synchronized with the first real-time image in time, and can replace the second real-time image with a large time difference and be added to the image set to be stitched. The reason for this is that image stitching usually requires different images to be as close as possible in acquisition time, otherwise there will be problems such as mismatch of target object position and deformation dislocation during stitching.

[0079] At the specific implementation level, a database or cache module is used to save the previous image set to ensure that the corresponding data source label can be quickly indexed when processing different batches of images, and the relevant historical images can be extracted. In order to further ensure the accuracy of time series matching, after retrieving the historical image, it will be checked again with the first real-time image according to the acquisition timestamp of the historical image. If the difference is still greater than the first time threshold, the historical image will be abandoned. Further, other possible candidate images can be searched or the image acquisition device can be marked as abnormal. In summary, by introducing the matching mechanism of data source labels and historical images, the data source selection in the image stitching process can be dynamically adjusted, and clear and complete surround stitching images can be generated in distributed, multi-camera surround acquisition, as well as under different acquisition conditions and network conditions. This flexible image supplementation mechanism improves the overall robustness and reliability, and avoids obvious errors in the entire surround image due to the late or early acquisition time of individual images. It provides vehicle drivers with more accurate visual information of the surrounding environment and improves driving safety.

[0080] It should be further explained here that if the acquisition time difference between the second real-time image and the historical image acquired by a certain image acquisition device and the first real-time image cannot meet the requirements of the first time threshold, at this time, a flexible image processing method can be used to ensure that the driver can obtain real-time surround view images as much as possible. In the first processing method, the second real-time image that does not meet the time synchronization requirements can be removed from the current image set, and other images that meet the time synchronization requirements are retained for subsequent stitching processes. This solution is intended to ensure that even if some image data is time-out, a usable surround view image can still be output for the driver's reference.

[0081] Another processing method is to save all the image data in the current image set, but not output it. Wait until all the second real-time images and / or the corresponding historical images meet the requirements of the first time threshold. At this time, all the image data will be spliced ​​according to the time synchronization, and finally a complete and accurate surround view image output will be generated for the driver to use. This method ensures the integrity and continuity of the surround view image. When selecting a specific data output process, it can be flexibly set according to the actual application scenario and needs. For example, in a traffic environment that requires a quick response, it is preferred to output an incomplete image for the driver's reference and make corrections at a later time; in a scenario where high image accuracy is required, it is chosen to wait for the image synchronization before outputting the complete image. This flexible data processing and output mechanism can adapt to the needs of different driving environments, improve driving safety and ensure overall real-time and stability.

[0082] In one embodiment, Figure 4 The flowchart of the method for performing image frame loss detection provided in the embodiment of the present application is a specific description of an implementation method of performing image frame loss detection in the above step S12. Figure 4 As shown, including:

[0083] S41, determining whether the acquisition time difference is less than or equal to a second time threshold;

[0084] S42, if all acquisition time differences are less than or equal to the second time threshold, determining that there is no image frame loss in the current image set;

[0085] S43, if any acquisition time difference is greater than the second time threshold, it is determined that image frame loss exists in the current image set, and an alarm message is sent to the vehicle central control terminal, so that the vehicle central control terminal can visualize the alarm message.

[0086] In this embodiment, in order to ensure the consistency and accuracy of image stitching, it is necessary to perform image frame loss detection on the acquisition of the current image set. Image frame loss refers to the situation that some images fail to be normally collected and transmitted to the system at a preset time interval due to data transmission delay, acquisition device failure or network instability, resulting in missing or incomplete image sequences. If the image frame loss problem is not detected and processed in time, it may cause the surround stitching image to appear blank, broken or misaligned, thereby affecting the display effect of the surround image and the decision accuracy of the vehicle driver. First, based on the acquisition timestamps of the first real-time image and each second real-time image in the current image set, the acquisition time difference between each real-time image is calculated and obtained. The acquisition time difference reflects the time interval between adjacent image frames. Under normal circumstances, the time interval should be kept within a certain range. If the time difference exceeds the preset range, it may indicate that there is an abnormality in the image acquisition process. Therefore, the second time threshold is introduced as a judgment criterion, and all acquisition time differences are compared one by one to determine whether there is an image frame loss situation.

[0087] During the detection process, it is determined in turn whether each acquisition time difference is less than or equal to the second time threshold. If all acquisition time differences are within the threshold range, it can be determined that there is no image frame loss in the current image set, indicating that all image frames are normally collected and transmitted to the system at the preset time interval, and data integrity is guaranteed. In this case, the next stage of image splicing and processing can be directly entered without additional image data. However, if it is found during the detection process that any acquisition time difference is greater than the second time threshold, it can be determined that there is an image frame loss problem in the current image set. At this time, it is necessary to send an alarm message to the vehicle central control terminal in a timely manner to prompt the driver that there is an abnormality in the image acquisition of the current surround view system. The alarm information may include the fault status of the image acquisition device, the time period when the frame loss occurs, and the possible scope of impact. In order to ensure that the alarm information can be noticed by the driver in time, it is visualized in the form of images and text through the central control terminal, for example, the frame loss status is highlighted on the screen in the form of a red flashing box, and a prompt sound is accompanied to remind the driver to pay attention. Through this image frame loss detection mechanism, abnormal conditions in the data acquisition process can be effectively identified before image splicing, and corresponding treatment measures can be taken in time. This not only improves the accuracy and stability of surround view image stitching, but also provides real-time status feedback to the driver, reducing safety risks caused by data anomalies. Especially in complex driving environments or severe weather conditions, the frame loss detection mechanism can effectively ensure overall reliability and real-time performance.

[0088] Here, a method for processing images after frame loss is given. For example, if the acquisition time difference between the second real-time image and the first real-time image exceeds the set time threshold, the image is considered to have frame loss. At this time, the image is removed from the set of images to be spliced. In this way, the lost image will not participate in the subsequent splicing process, avoiding the splicing error caused by image time asynchrony. For the remaining images, time synchronization processing will continue, and all valid images will be spliced ​​using the image splicing algorithm to generate a surround view image. It should be noted that although the frame-lost image does not participate in the splicing, the remaining images can still be correctly spliced ​​and displayed, ensuring that the surround view system can still provide visual information about some surrounding environment in the case of image frame loss, ensuring that the driver has sufficient reference basis during driving. At the same time, a fault warning message is sent to the vehicle central control terminal. After receiving the alarm message, the central control terminal will display the current surround view splicing image through a graphical interface, and attach relevant fault prompts to remind the driver to pay attention to the system status. This can not only ensure the real-time and stability of the surround view system, but also provide warning information in time when an abnormality occurs, further improving driving safety.

[0089] In one embodiment, Figure 5 The following is a flow chart of a method for obtaining a surround stitching image provided in an embodiment of the present application. It is a specific description of an implementation method of the above step S14. Based on the above embodiment, Figure 5 As shown, including:

[0090] S51, recording the second real-time image and / or the historical image in the set of images to be stitched, and the first real-time image as images to be processed;

[0091] S52, using an image segmentation algorithm to identify and obtain feature points of each image to be processed according to all the images to be processed;

[0092] S53, obtaining an image transformation matrix of each image to be processed according to the calibration parameters of the surround view system and the feature points of all the images to be processed;

[0093] S54, obtaining a corrected image corresponding to each image to be processed by adjusting parameters according to the image transformation matrix of all images to be processed;

[0094] S55, stitching the corrected images according to the correspondence between the feature points to obtain a surround stitching image, and sending the surround stitching image to the vehicle central control terminal so that the central control terminal can visualize and display the surround stitching image.

[0095] In this embodiment, the second real-time image and / or historical image in the image set to be stitched are included in the same processing flow together with the first real-time image to make full use of the diverse image information in time and source to generate a surround stitching image. At the implementation level, a list of images to be processed is usually allocated first, and all selected images (including the first real-time image, the second real-time image, and the historical image obtained from the previous image set) are uniformly placed in the list, and each image is assigned a unique identifier and corresponding metadata information (such as resolution, distortion correction coefficient, camera ID, etc.) for tracking and management in subsequent operations. When performing image segmentation algorithms on these images to be processed, the images are usually divided into more consistent areas based on technologies such as brightness differences, texture features, or edge detection. Optionally, an existing deep learning model can be used to identify the main structures or objects in the scene, thereby assisting in determining the initial distribution position of feature points. Feature points here refer to pixel areas that can have significant contrast in geometry or brightness, such as corners, cross-shaped textures, strong edge intersections, etc. Common feature point detection methods include SIFT (Scale-Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF) or AKAZE (Accelerated-KAZE). A large number of feature points can be detected in each image, and the information of these feature points (such as coordinates, scale, and directional features) can be recorded in the feature data structure. For example, for target objects with obvious boundaries such as road markings around vehicles, building edges, and vehicle outlines, they are extracted as feature points, and the corresponding coordinate information is annotated for each feature point.

[0096] Next, according to the calibration parameters of the surround view system and the detected feature point information, the image transformation matrix is ​​calculated and generated. In the specific implementation, the internal and external parameters of each image (internal parameters include focal length, optical center position and distortion coefficient, etc., and external parameters include the rotation and translation of the camera relative to the world coordinate system) are first input into the projection model calculation. Then, by matching the feature points between the images, the relative mapping relationship between the images is calculated, such as the homography matrix or the transformation matrix of other projection models, to ensure that the images can be mapped to the same coordinate system. The matrix generation process involves geometric transformation operations such as rotation, translation, and scaling, specifically including converting the coordinates of the feature points in the image into a coordinate system that matches the panoramic view of the vehicle. For example, for the image data collected by the camera installed on the left side of the vehicle, the rotation angle and translation distance of the image are calculated according to the installation angle and position of the camera, thereby generating a transformation matrix. If the images are not completely overlapping or the angle difference between different cameras is large, it is usually necessary to execute algorithms such as RANSAC (Random Sample Consensus) to exclude abnormal matching points to improve the accuracy of the transformation matrix.

[0097] When adjusting parameters, corrections are made for geometric and brightness differences respectively. Geometric correction includes further correction of distortion and perspective differences to ensure that there is no obvious distortion or misalignment when stitching the images; brightness correction unifies the brightness and color of each image based on the image histogram or statistical characteristics, such as using multi-channel gain or gamma correction methods to make the pixels in the overlapping area tend to transition smoothly in grayscale or chromaticity. This adjustment process can be performed on a global or local basis. The global approach usually applies the same brightness and color adjustment to the entire image; the local approach first detects the overlapping area of ​​the image and determines the color matching strategy of the adjacent area based on the statistical characteristics of the area to minimize the visible differences at the stitching seams. For example, when a vehicle collects images under different lighting conditions, the image may appear too bright or too dark due to changes in light intensity. Through parameter adjustment, the brightness of these images can be uniformly adjusted to the preset standard range to ensure that the surround stitching image will not have stitching faults or color differences due to brightness differences.

[0098] After all the corrected images are acquired, the images are stitched together based on the matching relationship of the feature points. For example, for the images captured by the cameras on the left and right front of the vehicle, the overlapping areas in the two images are identified, and the edges of the two areas are aligned through the image stitching algorithm so that they appear continuous in the stitched image. For the overlapping areas, the Alpha Blending or Seam Cutting algorithms can be used to eliminate the seams at the edges of multiple images. Alpha blending will perform linear or nonlinear weight distribution based on the distance between the pixel and the edge to achieve a smooth transition; automatic cutting will find a "stitching line" with the smallest difference for different overlapping areas to minimize visual inconsistencies. Ultimately, the stitched output surround view image can cover more perspectives of the vehicle's surrounding environment, and is presented in a panoramic or bird's-eye view on the central control terminal, allowing the driver to observe the composite scenes from different times or different cameras in a unified picture.

[0099] Through this embodiment, high-precision surround stitching based on multi-source images and multi-time information is achieved. For the driver, this stitching result can visualize a coherent surround view on the vehicle's central control terminal, providing more comprehensive visual information for blind spot detection, reversing assistance or other driving assistance functions. In addition, if there are higher requirements, it can be further combined with depth maps or geographic information (such as GPS data) to compensate for geometric errors between different perspectives, thereby obtaining a more accurate scene map around the vehicle.

[0100] Figure 6 A flow chart of a method for detecting whether a surround view system is working properly provided in an embodiment of the present application. After acquiring the current image set in the above step, the image acquisition device in the surround view system is detected to determine whether it is working properly. Based on the above embodiment, Figure 6 As shown, including:

[0101] S61, obtaining the number of image acquisition devices in the surround view system, and judging whether the preset image acquisition devices in the surround view system are working normally according to the number of images in the current image set;

[0102] S62, if the number of images is less than the number of image acquisition devices, reading and analyzing the data source tags carried by all images in the current image set to screen out faulty image acquisition devices that are not working properly;

[0103] S63, after determining that the abnormal time of the fault image acquisition device meets the third time threshold, sending the surround view stitching image carrying the fault warning information to the vehicle central control terminal, so that the vehicle central control terminal can visualize the surround view stitching image and the fault warning information.

[0104] In this embodiment, in order to ensure the normal operation of the surround view system and the integrity of image acquisition, it is first necessary to obtain the number of image acquisition devices preset in the surround view system. The surround view system is usually composed of multiple image acquisition devices deployed in the front, rear, left and right sides of the vehicle. Each image acquisition device is responsible for collecting image data from different angles of the vehicle to achieve 360-degree panoramic monitoring of the vehicle's surroundings. In actual applications, the number and installation positions of these image acquisition devices are usually fixed and have been determined in the design stage. Therefore, before performing image processing, it is first necessary to read and record the number of these preset image acquisition devices as a basis for verifying subsequent image acquisition data.

[0105] After obtaining the number of preset image acquisition devices, it will be determined whether the preset image acquisition devices in the surround view system are working properly according to the number of images in the current image set. Specifically, the current image set is a set of image data collected by each image acquisition device in the surround view system at the same time. Under normal circumstances, the number of images in the current image set should be consistent with the number of preset image acquisition devices. If it is found that the number of images in the current image set is less than the preset number of image acquisition devices, it indicates that one or more image acquisition devices may not work properly. At this time, the image data in the current image set will be further analyzed to determine the specific faulty image acquisition device. To this end, the data source tag carried by each image in the current image set will be read and analyzed. By analyzing these data source tags, it is possible to quickly filter out which image acquisition devices do not provide image data, thereby determining the specific faulty image acquisition device. For example, if it is detected that the image data from the left camera of the vehicle is missing in the current image set, it can be preliminarily determined that the image acquisition device on the left side of the vehicle may be faulty. The use of data source tags can not only improve the accuracy of fault location, but also provide a basis for subsequent fault detection and maintenance.

[0106] After the faulty image acquisition device is determined, it is also necessary to track and record the abnormal working time of the faulty device. If the abnormal working time of the faulty image acquisition device exceeds the preset third time threshold, the fault will be identified as a permanent fault, not just a short signal interruption or occasional error. The third time threshold can be set according to the specific application scenario and safety requirements, and is usually a shorter time period to ensure that the fault of the image acquisition device can be discovered and handled in time. For example, the third time threshold can be set to 10 seconds or 30 seconds to ensure that a response can be made in a short time after the image acquisition device fails. After determining that the abnormal working time of the faulty image acquisition device meets the third time threshold, a surround view spliced ​​image carrying fault warning information will be generated and sent to the central control terminal of the vehicle. The surround view spliced ​​image is a panoramic view spliced ​​by normal image data in the current image set, and the fault warning information is used to remind the driver that a certain image acquisition device is not working properly. For example, the location of the faulty image acquisition device can be highlighted in the surround view spliced ​​image, or a text prompt message can be popped up on the central control terminal to inform the driver of the specific fault situation. This visual display method can intuitively convey fault information to the driver, helping him to take corresponding safety measures in a timely manner during driving.

[0107] In order to improve the readability and prominence of fault warning information, multiple display modes can also be used on the central control terminal. For example, the location of the faulty image acquisition device can be identified by color changes, and the fault area can be highlighted in red or yellow; or a scrolling text can be displayed on the edge of the surround view spliced ​​image to remind the driver to pay attention to the fault of the image acquisition device. In addition, the communication effect of the fault warning information can be further enhanced by various means such as sound prompts or vibration feedback. Through this multi-modal warning information display method, the driver's attention can be effectively improved and the driving safety problems caused by the failure of the image acquisition device can be reduced. In summary, by obtaining the number of image acquisition devices in the surround view system, judging the working status of the image acquisition device, analyzing the data source label to filter the faulty device, tracking the abnormal working time, and sending the fault warning information to the central control terminal, the real-time monitoring and fault warning of the state of the image acquisition device of the surround view system are realized. This fault detection and warning mechanism can not only ensure the integrity of the image data of the surround view system, but also provide fault information to the driver in time when the image acquisition device fails, help take safety measures, and thus improve the overall driving safety performance of the vehicle.

[0108] In a specific embodiment, the current image set acquired by the surround view system is received through a preset wireless home digital interface.

[0109] In this embodiment, the Wireless Home Digital Interface (WHDI) is a commonly used high-bandwidth wireless communication technology that can support real-time transmission of high-quality image and video data. Through the preset wireless home digital interface, the current image set collected by the surround view system can be wirelessly transmitted to the vehicle or other processing terminals. When deployed, the interface is usually connected to a gateway device inside or outside the vehicle to achieve high-bandwidth, low-latency transmission of multiple image data. After acquiring the surround view image, the image acquisition device will first perform preliminary compression or packaging processing on the image, and then send the data out in the form of a digital stream through the interface. At the same time, the received image packet is parsed to extract the acquisition timestamp and other related metadata of each image for subsequent image synchronization and splicing operations. Since this method can reduce the limitations of traditional wired connections in spatial layout and routing, the wireless home digital interface is more flexible when deployed on the vehicle and can reduce the hidden dangers of cable clutter or susceptibility to physical wear. Through such interface deployment, the vehicle can continuously acquire and update the real-time image set collected by the surround view system during driving, providing a stable and high-speed image data transmission channel for blind spot detection or other advanced driving assistance functions.

[0110] In practical applications, in order to achieve stability and security of data transmission, when receiving image data through the wireless home digital interface, an encrypted transmission protocol can be used to prevent the image data from being illegally intercepted or tampered with during transmission. Common encryption protocols include AES (Advanced Encryption Standard) and SSL (Secure Sockets Layer). These encryption protocols can effectively protect the integrity and confidentiality of image data and ensure that the image data transmitted to the vehicle's central control processing unit is reliable and accurate. In addition, in order to improve the anti-interference ability of data transmission, frequency hopping technology or channel switching technology can be used to avoid interference from other wireless devices and ensure the continuity and stability of data transmission. Receiving the current image set through the wireless home digital interface not only improves the transmission efficiency of image data, but also improves the overall stability and adaptability of the system. In practical applications, the use of wireless interfaces can reduce maintenance costs and reduce failures caused by aging or damage of cables. At the same time, the communication parameters of the wireless interface, such as channel frequency, bandwidth, and transmission power, can be dynamically adjusted according to the actual use scenario and communication environment of the vehicle to adapt to different network environments and transmission requirements, thereby ensuring efficient transmission of image data and reliable operation of the surround view system.

[0111] Figure 7 A schematic diagram of the structure of the image processing device provided in this application, such as Figure 7 As shown, the image processing device 70 provided in this embodiment includes:

[0112] The information acquisition module 701 is used to receive the current image set acquired by the surround view system, and according to the current image set, record the image with the earliest acquisition time as the first real-time image, and record the image with a later acquisition time than the first real-time image as the second real-time image;

[0113] The frame loss detection module 702 is used to obtain the acquisition time difference between the first real-time image and each second real-time image based on the current image set and according to the acquisition time stamps of the first real-time image and all the second real-time images, and perform image frame loss detection on the current image set according to all the acquisition time differences;

[0114] The time synchronization module 703 is used to obtain, after determining that there is no image frame loss in the current image set, a set of images to be stitched corresponding to the acquisition timestamp according to the acquisition timestamp of the first real-time image through time synchronization; wherein the set of images to be stitched includes the second real-time image and / or the historical images in the previous image set;

[0115] The image stitching module 704 is used to stitch the second real-time image and / or the historical image in the image set to be stitched, and the first real-time image, to obtain a stitched image, and complete image processing.

[0116] The image processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail in this embodiment.

[0117] Figure 8 The schematic diagram of the structure of the vehicle provided in this application. Figure 8 As shown, the vehicle 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the vehicle 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0118] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0119] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0120] In a specific embodiment, Fig. 9 FIG. 1 is a schematic diagram of the structure of a vehicle based on wireless transmission according to an embodiment of the present application. Fig. 9 As shown, it includes a camera, a wireless data sending module, a wireless data receiving module, a data synchronization module and a display control module.

[0121] The cameras are arranged at the front, rear, left and right of the vehicle to collect real-time image data of the environment around the vehicle. The cameras send the collected image data to the wireless data receiving module in the form of wireless signals through the preset wireless data transmission module. The wireless data transmission module supports multi-channel parallel transmission to ensure that the image data of cameras in different positions can be sent without interference.

[0122] The wireless data receiving module is used to receive image data from multiple wireless data sending modules and temporarily store the received image data in the cache space for subsequent processing by the data synchronization module. The cache space is designed as a ring queue structure, which can dynamically store real-time transmitted image data to avoid data loss due to differences in data processing speed.

[0123] The data synchronization module includes a time synchronization unit and a frame loss detection unit. The time synchronization unit performs time alignment on the image data from different cameras based on the acquisition timestamp in the image data to ensure that the image data processed subsequently is consistent in the time dimension. The frame loss detection unit detects whether there is image frame loss by calculating the time interval of each image data. If a camera's data is found to have frame loss, the frame loss image of the camera will be automatically removed, and the fault information will be recorded for subsequent processing.

[0124] The display and control module includes a surround view splicing display unit and a fault alarm display unit. The surround view splicing display unit generates a surround view splicing image based on the synchronized image data through an image splicing algorithm, and sends the generated surround view image to the vehicle's central control screen for visual presentation, so that the driver can intuitively understand the environmental conditions around the vehicle. If the system detects image frame loss or camera failure, the fault alarm display unit synchronizes the fault information to the central control screen to alert the driver to possible incomplete surround view images.

[0125] This embodiment can obtain surround view images around the vehicle in real time and ensure the time synchronization and integrity of image data during wireless transmission. Even if there is a frame loss of image data, it can still generate reliable surround view images and provide corresponding alarm information, thereby improving the reliability of the surround view system and driving safety.

[0126] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0127] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0128] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0129] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0130] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0131] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0132] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0133] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0136] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0137] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0138] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image processing method based on a surround view system, characterized in that: include: Receiving a current image set acquired by the surround view system, and based on the current image set, recording an image with the earliest acquisition time as a first real-time image, and recording an image acquired later than the first real-time image as a second real-time image; Based on the current image set, acquiring an acquisition time difference between the first real-time image and each of the second real-time images according to acquisition timestamps of the first real-time image and all the second real-time images, and performing image frame loss detection on the current image set according to all the acquisition time differences; After determining that there is no image frame loss in the current image set, acquiring a set of images to be stitched corresponding to the acquisition timestamp according to the acquisition timestamp of the first real-time image through time synchronization; wherein the set of images to be stitched includes the second real-time image and / or historical images in the previous image set; Perform surround image stitching on the second real-time image and / or the historical image in the set of images to be stitched, and the first real-time image, to obtain a surround stitching image, and complete image processing.

2. The method according to claim 1, characterized in that The acquiring, according to the acquisition timestamp of the first real-time image, a set of images to be stitched corresponding to the acquisition timestamp through time synchronization includes: According to the acquisition time difference between the first real-time image and the second real-time image, determining whether the acquisition time difference is less than or equal to a first time threshold; If the acquisition time difference is less than or equal to the first time threshold, the second real-time image is stored in the set of images to be stitched, so as to obtain the set of images to be stitched.

3. The method according to claim 2, characterized in that The acquiring, according to the acquisition timestamp of the first real-time image, a set of images to be stitched corresponding to the acquisition timestamp through time synchronization also includes: If the acquisition time difference is greater than the first time threshold, acquiring a data source tag carried by the second real-time image according to the second real-time image corresponding to the acquisition time difference; Acquire a previous image set acquired by the surround view system, and acquire a historical image corresponding to the data source label from the previous image set; According to the acquisition timestamp of the first real-time image and the acquisition timestamp of the historical image, after determining that the acquisition time difference between the first real-time image and the historical image is less than or equal to a first time threshold, the historical image is stored in the set of images to be stitched to obtain the set of images to be stitched.

4. The method according to claim 1, characterized in that The performing image frame loss detection on the current image set according to all the acquisition time differences includes: Determine whether the acquisition time difference is less than or equal to a second time threshold; If all the acquisition time differences are less than or equal to the second time threshold, it is determined that there is no image frame loss in the current image set; or, If any of the acquisition time differences is greater than the second time threshold, it is determined that image frame loss exists in the current image set, and an alarm message is sent to the vehicle central control terminal so that the vehicle central control terminal can visualize the alarm message.

5. The method according to any one of claims 1 to 3, characterized in that: The step of performing surround image stitching on the second real-time image and / or the historical image in the set of images to be stitched, and the first real-time image, to obtain a surround stitched image includes: Recording the second real-time image and / or the historical image in the set of images to be stitched, and the first real-time image, as images to be processed; According to all the images to be processed, an image segmentation algorithm is used to identify and obtain feature points of each image to be processed; Acquire an image transformation matrix of each image to be processed according to the calibration parameters of the surround view system and the feature points of all the images to be processed; According to the image transformation matrix of all the images to be processed, a corrected image corresponding to each image to be processed is obtained by adjusting parameters; According to the correspondence between the feature points, the corrected images are stitched together to obtain a surround stitching image, and the surround stitching image is sent to the vehicle central control terminal so that the central control terminal can visualize the surround stitching image.

6. The method according to claim 1, characterized in that After receiving the current image set acquired by the surround view system, the method further includes: Acquire the number of image acquisition devices in the surround view system, and determine whether the preset image acquisition devices in the surround view system work normally according to the number of images in the current image set; If the number of images is less than the number of image acquisition devices, reading and analyzing the data source tags carried by all images in the current image set to screen out faulty image acquisition devices that are not working properly; After determining that the abnormal time of the fault image acquisition device meets the third time threshold, the surround view stitching image carrying the fault warning information is sent to the vehicle central control terminal, so that the vehicle central control terminal can visualize the surround view stitching image and the fault warning information.

7. The method according to any one of claims 1 to 4, characterized in that: The current image set acquired by the surround view system is received through a preset wireless home digital interface.

8. An image processing device, characterized in that: include: An information acquisition module, configured to receive a current image set acquired by a surround view system, and based on the current image set, record an image acquired earliest as a first real-time image, and record an image acquired later than the first real-time image as a second real-time image; a frame loss detection module, configured to obtain, based on the current image set and according to acquisition timestamps of the first real-time image and all the second real-time images, an acquisition time difference between the first real-time image and each of the second real-time images, and perform image frame loss detection on the current image set according to all the acquisition time differences; A time synchronization module, configured to, after determining that there is no image frame loss in the current image set, obtain, according to the acquisition timestamp of the first real-time image, a set of images to be stitched corresponding to the acquisition timestamp through time synchronization; wherein the set of images to be stitched includes the second real-time image and / or historical images in the previous image set; The image stitching module is used to stitch the second real-time image and / or the historical image in the set of images to be stitched, and the first real-time image, to obtain a stitched image and complete image processing.

9. A vehicle, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images