A method and system for multi-path image synchronous display for 360 panorama system

Abstract

The application provides a method and system for multi-channel image synchronous display of a 360 panoramic system, which can ensure the synchronization of multi-channel images, reduce the deviation of panoramic image splicing, and ensure the display effect of 360 panoramic images, and comprises the following steps: receiving multi-channel image data, respectively judging whether the image data of each channel meets the time delay requirement, temporarily buffering the image data meeting the time delay requirement, judging whether the image data of all channels meeting the time delay requirement is received, after the image data of all channels is received, judging whether the image data of all channels meets the synchronization requirement, packaging and sending the image data meeting the synchronization requirement to an image processing module for processing, and splicing to obtain 360 panoramic images.

Description

Technical Field

[0001] This invention relates to the field of 360° panoramic system technology for vehicles, and specifically to a method and system for synchronous display of multi-channel images in a 360° panoramic system. Background Technology

[0002] A 360-degree panoramic system is an in-vehicle device that primarily provides visual monitoring of the vehicle's surroundings to facilitate driving and parking, avoiding blind spots and inconvenience. The system uses four cameras located at the front, sides, and rear of the vehicle to monitor the surrounding environment, providing the driver with a 360-degree panoramic view. This view is generated by stitching together images from multiple cameras. To ensure the display quality, authenticity, and validity of the 360-degree panoramic image, the image data collected by the multi-channel cameras is typically required to be synchronized as much as possible.

[0003] In existing technologies, the main methods to ensure synchronization include hardware synchronization and software synchronization. Using hardware GPIO to directly synchronize camera images increases hardware design costs and does not add fault handling methods after image loss. The software synchronization method requests a thread to acquire images for each channel, and acquires four channels of original images through thread blocking. There is no constraint on the timestamps of single and multi-frame images, which may lead to excessive delay in displaying images, resulting in poor stitching of 360-degree panoramic images, poor display effects, and poor realism. Furthermore, the software synchronization method does not have a fault handling strategy for the loss of a certain channel of images. The channel with the lost image is blocked indefinitely, making the 360-degree panoramic function unusable. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a method and system for synchronous display of multi-channel images in a 360° panoramic system. This method ensures the synchronization of multi-channel images, reduces panoramic image stitching deviations, and guarantees the display effect of 360° panoramic images.

[0005] The technical solution is as follows: A method for synchronous display of multiple images in a 360° panoramic system, comprising the following steps:

[0006] Receive multiple channels of image data, determine whether each channel of image data meets the latency requirements, and temporarily buffer the image data that meets the latency requirements;

[0007] Determine whether image data from all channels that meet the latency requirements has been received;

[0008] After receiving image data from all channels, it determines whether the image data from all channels meets the synchronization requirements. Image data that meets the synchronization requirements is packaged and sent to the image processing module for processing, and then stitched together to obtain a 360-degree panoramic image.

[0009] Furthermore, image data that does not meet the delay requirements is discarded, and image data that does not meet the synchronization requirements is also discarded.

[0010] Furthermore, the channel number of the image data is used to determine whether all channels of image data have been received.

[0011] Furthermore, if image data for a certain channel is not received, it is determined whether the corresponding channel meets the black-out requirement. For channels that meet the black-out requirement, a solid color image is used to replace the image data of the corresponding channel.

[0012] Furthermore, determine whether the image data for each channel meets the latency requirements, specifically as follows:

[0013] The time interval between the timestamp of the received image data and the timestamp of the acquired image is compared with a set delay threshold. If the time interval is less than the delay threshold, the delay requirement is met; otherwise, it is not.

[0014] Furthermore, determine whether the image data of all channels meets the synchronization requirements, specifically as follows:

[0015] Obtain the timestamps of the received image data from all channels, calculate the average reception time T, and let Tx be the timestamp corresponding to each channel. If abs(Tx-T)≤T 同步 Then the corresponding image data meets the synchronization requirements, where abs represents the absolute value of the data, and T 同步 The set synchronization threshold.

[0016] Further, determine whether the corresponding channel meets the blackout requirements, specifically as follows:

[0017] If no image data for a certain channel is received within the set blackout threshold time period, a black image is used to replace the image data for the corresponding channel. If an image is received within the blackout threshold time period, the image for the corresponding channel is restored.

[0018] A system for synchronous display of multiple images in a 360° panoramic system, comprising:

[0019] The image acquisition module is used to acquire multiple image data streams.

[0020] The latency determination module is used to receive multiple channels of image data, determine whether each channel of image data meets the latency requirements, and temporarily cache the image data that meets the latency requirements in the local data queue.

[0021] The multi-channel detection module is used to determine whether image data from all channels that meet the latency requirements have been received;

[0022] The synchronization determination module is used to determine whether the image data of all channels meets the synchronization requirements after receiving the image data of all channels.

[0023] The image processing module receives multiple image data that meet the synchronization requirements as determined by the synchronization determination module, and stitches the images to obtain a 360-degree panoramic image.

[0024] A computer device includes a processor, a memory, and a program; the program is stored in the memory, and the processor invokes the program stored in the memory to execute the above-described method for synchronous display of multiple images in a 360° panoramic system.

[0025] A computer-readable storage medium for storing a program for executing the above-described method for synchronous display of multiple images in a 360° panoramic system.

[0026] The present invention has the following advantages:

[0027] The present invention provides a method for synchronous display of multiple images in a 360° panoramic system. This method incorporates a mechanism to determine whether the image data of each channel meets a latency requirement. The latency requirement is determined by comparing the time interval between the timestamp of received image data and the timestamp of image acquisition with a set latency threshold. This latency threshold parameter helps assess the real-time performance of the 360° panoramic system. Simultaneously, the invention also determines whether the multi-channel image data meets synchronization requirements by increasing the synchronization threshold for multiple cameras. This parameter reduces panoramic image stitching deviations. Furthermore, an image blackout time parameter is added. If an image channel cannot be received and image data from that channel is not received within a set blackout threshold time period, a black image is used to replace the corresponding channel's image data. This ensures that multiple images function correctly, data is transmitted to the image processing module, and the 360° panoramic system is usable. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating the steps of a method for synchronous display of multiple images in a 360° panoramic system according to an embodiment of the present invention.

[0029] Figure 2 A flowchart of a method for synchronous display of multi-channel images in a 360-degree panoramic system according to an embodiment of the present invention;

[0030] Figure 3 This is a block diagram of a system for synchronous display of multiple images in a 360-degree panoramic system according to one embodiment of the present invention.

[0031] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0032] In existing technologies, a synchronization request is initiated from the GPIO on the SOC to the camera, and the original images from the four cameras are then input to the SOC. Another approach is to allocate a thread to acquire the image for each image channel, using a thread blocking method to acquire the four original images. Using hardware to synchronize camera images increases equipment costs, and synchronizing the four images using thread blocking results in poor synchronization of the four images, leading to incomplete stitching of the 360-degree panoramic image. To address this, the present invention was developed as a method for multi-channel image synchronization display in a 360-degree panoramic system. This invention is based on 360-degree panoramic imaging, and the project requirements are that the single-channel image meets the latency requirements, the four cameras need to meet the synchronization time requirements, and the 360-degree panoramic functionality can be used even when image faults exist.

[0033] See Figure 1 , 2 The present invention provides a method for simultaneous display of multiple images in a 360° panoramic system, comprising the following steps:

[0034] Step 1: Continuously receive image data from 4 cameras. At image frame intervals, determine whether the image data of each camera meets the latency requirement. If the latency requirement is met, temporarily buffer it in the local data queue; otherwise, discard the image frame.

[0035] Step 2: Determine whether all 4 channels of image data that meet the latency requirements have been received. If less than 4 images are received, or if image data for a certain channel is not received, determine whether the corresponding channel meets the blackout requirement. For channels that meet the blackout requirement, use a solid color image to replace the image data of the corresponding channel. If a frame of image is received within the blackout period, the image of the corresponding channel is immediately restored.

[0036] Step 3: If 4 channels of images are received, after receiving the image data of all channels, determine whether the image data of all channels meets the synchronization requirements. Pack the image data that meets the synchronization requirements and send it to the image processing module for processing. In this embodiment, the image processing module is implemented through the GPU module. If the synchronization requirements are met, the 4 channels of images are packed and sent to the GPU module to stitch together to obtain a 360-degree panoramic image. If the image of a certain channel does not meet the synchronization requirements, the image data of the corresponding channel is discarded.

[0037] In one specific embodiment of the present invention, it is determined whether the image data of each channel meets the delay requirement. Specifically, the time interval between the timestamp of receiving a certain image data channel and the timestamp of acquiring the image can be compared with a set delay threshold. If the time interval is less than the delay threshold, the delay requirement is met; otherwise, it is not met. The delay threshold can be set by the user to 500ms or 300ms. If this time is exceeded, the frame image is discarded.

[0038] In one specific embodiment of the present invention, when receiving multi-channel image data, the channel number of the image data is obtained simultaneously. The channel number of the image data can then be used to determine whether all channels of image data have been received.

[0039] In one specific embodiment of the present invention, determining whether the image data of all channels meets the synchronization requirements is performed as follows:

[0040] Obtain the timestamps of the received image data from all channels, calculate the average reception time T, and let Tx be the timestamp corresponding to each channel. If abs(Tx-T)≤T 同步 Then the corresponding image data meets the synchronization requirements, where abs represents the absolute value of the data, and T 同步 The set synchronization threshold is T. In this embodiment, T is set to... 同步 Given 50ms, calculate the average time stamp T of the 4-channel images. The time stamp corresponding to each channel is Tx. If abs(Tx-T)<=50ms, it means that the timing of the 4-channel images meets the synchronization requirements.

[0041] In one specific embodiment of the present invention, it is determined whether the corresponding channel meets the blackout requirement. Specifically, the following steps are performed: if no image data of a certain channel is received within the set blackout threshold time period, a black image is used to replace the image data of the corresponding channel. The time during which no image is received for a certain channel within X seconds is used as the blackout threshold. For example, it can be configured as 1S or 2S. If this time is exceeded, the image of the corresponding channel is set to black. Of course, it can also be set to a solid color image such as dark blue, brown, or gray.

[0042] This embodiment considers the timing parameters of a single camera, the synchronization parameters of four cameras, and ensuring the normal operation of other cameras in case of camera failure. It incorporates a method to determine whether the image data of each channel meets the latency requirements. The latency requirement is determined by comparing the time interval between the timestamp of received image data and the timestamp of the acquired image with a set latency threshold. This latency threshold parameter can be used to determine the real-time performance of the 360-degree panoramic system. Simultaneously, this invention determines whether the image data of multiple channels meets the synchronization requirements by increasing the synchronization threshold for multiple cameras. This parameter can reduce panoramic image stitching deviation. Furthermore, an image blackout time parameter is added. If an image channel cannot be received, and image data from that channel is not received for a set blackout threshold period, a black image is used to replace the corresponding channel's image data, ensuring that multiple images can function normally and that data can be correctly sent to the image processing module, ensuring that the 360-degree panoramic system is partially usable.

[0043] See Figure 3 In an embodiment of the present invention, a system for simultaneous display of multiple images in a 360-degree panoramic system is also provided, comprising:

[0044] Image acquisition module 1 is used to acquire multiple image data streams;

[0045] The delay determination module 2 is used to receive multiple channels of image data, determine whether each channel of image data meets the delay requirements, and temporarily cache the image data that meets the delay requirements in the local data queue.

[0046] Multi-channel detection module 3 is used to determine whether image data from all channels that meet the latency requirements have been received;

[0047] Synchronization determination module 4 is used to determine whether the image data of all channels meets the synchronization requirements after receiving the image data of all channels;

[0048] Image processing module 5 receives multi-channel image data that meets the synchronization requirements as determined by the synchronization determination module, and stitches the images to obtain a 360-degree panoramic image.

[0049] In one embodiment of the present invention, the capture module of the SOC receives raw images from four cameras. After receiving each frame of raw image, the channel number and timestamp are recorded and sent to the system in this embodiment. After receiving the four independent raw images, the system combines the four independent images into a composite image based on their channel numbers, timestamps, and user input parameters, and sends it to the GPU for processing. At this time, the GPU can directly use the composite image to perform 360-degree panoramic function. Since the composite image meets the time synchronization requirements, the 360-degree panoramic image has low latency and good stitching. At the same time, even if the camera malfunctions, it can still ensure that part of the 360-degree panoramic function works. The composite image is simply four independent images packaged into one data packet, and this data packet includes information such as the four raw images, channel numbers, and timestamps.

[0050] After receiving an image frame, the system of this invention determines whether the single-channel image delay requirement is met based on the current system time and the timestamp of the image acquisition. When the system receives four images, it determines whether the channel number and timestamp of the four images meet the synchronization requirement of the four images. When the camera or wiring harness fails and the software synchronization module receives fewer than four images, it fills the missing images with black images to ensure that some functions of the 360-degree panoramic system can still be used.

[0051] In this embodiment, SoC, also known as system-on-a-chip, means that it is a product, an integrated circuit with a specific purpose, which contains a complete system and all the embedded software.

[0052] GPU stands for Graphics Processing Unit, which is a microprocessor specifically designed to perform image and graphics-related computations in personal computers, game consoles, and embedded devices.

[0053] The advantages of this invention are:

[0054] 1. The synchronization module and GPU module run separately, reducing the coupling between software modules, making the functions of the software modules clear and facilitating software module portability; in multi-core CPUs, they can run on separate cores, reducing the load on individual CPUs;

[0055] 2. Add a single-channel image delay parameter, which can be used to determine the real-time performance of the system;

[0056] 3. Added a synchronization time parameter for 4 cameras. Configuring this parameter can reduce panoramic image stitching deviation.

[0057] 4. Add an image blackout time parameter to ensure that all four images can work normally and data can be sent to the GPU normally when one image channel cannot be received, thus ensuring that the 360-degree panoramic system can be used.

[0058] In an embodiment of the present invention, a computer device is also provided, comprising: a processor, a memory, and a program;

[0059] The program is stored in memory, and the processor calls the program stored in memory to execute the above-described method for synchronous display of multiple images in a 360-degree panoramic system.

[0060] The computer device can be a terminal, and its internal structure diagram can be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for synchronous display of multiple images for a 360° panoramic system. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0061] Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory stores programs, and the processor executes these programs after receiving execution instructions.

[0062] A processor can be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. The processor can also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor.

[0063] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0064] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program for executing the above-described method for synchronous display of multiple images in a 360-degree panoramic system.

[0065] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, computer devices, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] Embodiments of the present invention are described with reference to flowchart illustrations of methods, computer apparatuses, or computer program products according to embodiments of the invention. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart.

[0067] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in the flowchart.

[0068] The above provides a detailed description of the method, system, computer device, and computer-readable storage medium for synchronous display of multi-channel images in a 360° panoramic system provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for synchronous display of multiple images in a 360° panoramic system, characterized in that, Includes the following steps: Receive multiple channels of image data, determine whether each channel of image data meets the latency requirements, and temporarily buffer the image data that meets the latency requirements; Determine whether image data from all channels that meet the latency requirements has been received; After receiving image data from all channels, it is determined whether the image data from all channels meets the synchronization requirements. The image data that meets the synchronization requirements is packaged and sent to the image processing module for processing, and then stitched together to obtain a 360-degree panoramic image. The channel number of the image data is used to determine whether image data from all channels has been received. Get the timestamps of all received image data from all channels, calculate the average receiving time, and if the absolute value of the difference between the timestamp of the corresponding channel and the average receiving time is less than or equal to the set synchronization threshold, the image data of the channel is considered to meet the synchronization requirements. Image data that does not meet the latency requirements is discarded; image data that does not meet the synchronization requirements is also discarded. If image data for a certain channel is not received, determine whether the corresponding channel meets the black setting requirement. For channels that meet the black setting requirement, replace the image data of the corresponding channel with a solid color image.

2. The method for simultaneous display of multiple images in a 360° panoramic system according to claim 1, characterized in that: To determine whether the image data for each channel meets the latency requirements, the following steps are performed: The time interval between the timestamp of the received image data and the timestamp of the acquired image is compared with a set delay threshold. If the time interval is less than the delay threshold, the delay requirement is met; otherwise, it is not.

3. The method for simultaneous display of multiple images in a 360° panoramic system according to claim 1, characterized in that: To determine whether the corresponding channel meets the blackout requirement, execute the following steps: If no image data for a certain channel is received within the set blackout threshold time period, a black image is used to replace the image data for the corresponding channel. If an image is received within the blackout threshold time period, the image for the corresponding channel is restored.

4. A system for simultaneous display of multiple images in a 360° panoramic system, characterized in that, include: The image acquisition module is used to acquire multiple image data streams. The latency determination module is used to receive multiple channels of image data, determine whether each channel of image data meets the latency requirements, and temporarily cache the image data that meets the latency requirements in the local data queue. The multi-channel detection module is used to determine whether image data from all channels that meet the latency requirements has been received. It determines whether image data from all channels has been received by using the channel number of the image data. The synchronization determination module is used to determine whether the image data of all channels meets the synchronization requirements after receiving the image data of all channels. The image processing module receives multi-channel image data that meets the synchronization requirements as determined by the synchronization determination module, stitches the images to obtain a 360-degree panoramic image, obtains the timestamps of the image data received from all channels, calculates the average receiving time, and if the absolute value of the difference between the timestamp corresponding to a channel and the average receiving time is less than or equal to a set synchronization threshold, then the image data of the channel is considered to meet the synchronization requirements. The delay determination module discards image data that does not meet the delay requirements, and the synchronization determination module discards image data that does not meet the synchronization requirements. If the multi-channel detection module does not receive image data for a certain channel, it determines whether the corresponding channel meets the blackening requirement. For channels that meet the blackening requirement, a solid color image is used to replace the image data of the corresponding channel.

5. A computer device comprising: It includes a processor, a memory, and a program; the program is stored in the memory, and the processor calls the program stored in the memory to execute the method for synchronous display of multiple images for a 360-degree panoramic system as described in any one of claims 1 to 3.

6. A computer-readable storage medium for storing a program, the program being executed by a processor as described in any one of claims 1 to 3, for the method of simultaneous display of multiple images in a 360° panoramic system.

Images