Panoramic look-around cold-start image output method and device and computer readable storage medium

Abstract

The invention provides a panoramic look-around cold start image output method and device and a computer readable storage medium, and the method comprises the steps: initializing an image collection device and a data transmission link of a vehicle in a kernel start or kernel initialization process of a Linux system, enabling the initialization of a camera to be advanced, and enabling the image collection device and the data transmission link to be more accurate. The problem of delay caused by initialization after the current service is started is solved. And on the basis of the service configuration file of the Linux system, system services are started according to a preset sequence, so that the resources can be called and used in time in the AVM drawing process, and drawing delay caused by resource waiting is avoided. The data of the image acquisition equipment is received and processed through the AVM application, the panoramic image is generated, and the panoramic image is displayed through the display of the cabin equipment of the vehicle, so that the time from the cold start of the vehicle to the display of the panoramic image is shortened, and the panoramic look-around cold start quick drawing is realized on the premise of not increasing the hardware cost.

Description

Technical Field

[0001] This application relates to the field of driving assistance technology, and particularly to a panoramic surround cold start image output method, device, and computer-readable storage medium. Background Art

[0002] An in-vehicle AVM (AroundView Monitor) captures images by installing multiple (usually four) ultra-wide-angle fish-eye lenses around the vehicle, and then uses special algorithms to correct the distortion and stitch these images to form a panoramic image around the vehicle. Such a system can provide a blind-spot-free driving view, helping the driver observe the environment around the vehicle more safely and easily when parking and driving. Currently, AVM is basically a standard configuration in the market.

[0003] For traditional parking controllers, the functions are single, the software and hardware systems are relatively simple, and the startup time is relatively short. For high-order intelligent driving platforms, in order to meet the requirements of high-order parking and driving intelligent driving functions, the software and hardware systems are relatively complex, and the startup time is usually more than 15 seconds, resulting in a long waiting time for users to use the AVM function after cold starting the vehicle.

[0004] Therefore, there is an urgent need for a panoramic surround cold start fast image output solution for high-order intelligent driving platforms. Summary of the Invention

[0005] To overcome the problems existing in the related art, this specification provides a panoramic surround cold start image output method, device, and computer-readable storage medium.

[0006] According to the first aspect of the embodiments of this specification, a panoramic surround cold start image output method is provided. The method includes:

[0007] Applied to an intelligent driving controller, the method includes:

[0008] Power on the MCU in the intelligent driving controller to wake up the SoC in the intelligent driving controller, so that the SoC loads the bootloader and boots the startup and initialization of the kernel of the Linux system;

[0009] During the startup or initialization of the kernel of the Linux system, initialize the vehicle's image acquisition device and data transmission link, and configure the data transmission channel between the image acquisition device and the Linux system;

[0010] Based on the service configuration file of the Linux system, start system services in a preset order. The system services include the panoramic surround system AVM application and Camera service;

[0011] Receive and process the data of the image acquisition device through the AVM application to generate a panoramic image; the data of the image acquisition device is obtained by the Camera service through the data transmission channel and then sent to the AVM application;

[0012] Display the panoramic image through the display of the vehicle's cockpit device.

[0013] According to a panoramic surround cold start image generation method provided by the present application, the method further includes:

[0014] Establish a communication connection between the MCU and the deserialiser in the intelligent driving controller, and the deserialiser is connected to the SoC;

[0015] After the MCU is started, initialize the vehicle's image acquisition device and data transmission link based on the MCU.

[0016] According to a panoramic surround cold start image generation method provided by the present application, the system service includes the systemd device trigger service, and the systemd device trigger service includes subsystem and device.

[0017] The service configuration file of the Linux system includes:

[0018] Configure the startup order of the subsystem and the device based on the dependency relationship between the Xorg service related to AVM image generation and the device.

[0019] Among them, the startup priority of the device is greater than that of the subsystem.

[0020] According to a panoramic surround cold start image generation method provided by the present application, the system service further includes a mounting service for each partition of the disk, so that the Linux system can access and use the files and data on each partition.

[0021] The service configuration file of the Linux system includes:

[0022] Based on the correlation between each partition of the disk and the AVM application, determine the mounting order for mounting each partition of the disk according to the degree of correlation.

[0023] Among them, the mounting order is that the mounting priorities of multiple partitions that have a direct correlation with the AVM application are greater than other partitions of the disk.

[0024] According to a panoramic surround cold start image generation method provided by the present application, multiple partitions that have a direct correlation with the AVM application are mounted in parallel;

[0025] and / or

[0026] Reduce the capacity of multiple partitions that have a direct correlation with the AVM application.

[0027] A panoramic surround view cold start image output method provided by the present application, the service configuration file based on the Linux system includes:

[0028] After the mounting is completed, immediately start the AVM application and the Camera service.

[0029] A panoramic surround view cold start image output method provided by the present application, the system service includes a data distribution service,

[0030] The service configuration file based on the Linux system includes:

[0031] The Camera service distributes the data of the image acquisition device to the AVM application through the dynamic multicast mechanism.

[0032] A panoramic surround view cold start image output method provided by the present application, the dynamic multicast mechanism includes an image consumer and an image producer, the image consumer includes the AVM application and other auxiliary function modules, and the image producer is the Camera service,

[0033] The method further includes:

[0034] In the case of multiple image consumers, the Camera Service distributes image data to all registered consumers in parallel through the dynamic multicast mechanism.

[0035] A panoramic surround view cold start image output method provided by the present application, the dynamic multicast mechanism includes:

[0036] When the Camera Service starts, create a dynamic multicast group and open a registration channel for image consumers to access;

[0037] The AVM application registers as the primary consumer to the multicast group, triggering the Camera Service to perform data transmission;

[0038] Other consumers perform asynchronous registration after the AVM application registration is completed, and dynamically join the multicast group to receive the data stream.

[0039] A panoramic surround view cold start image output method provided by the present application, the method further includes:

[0040] When within the set time range of the startup of the SoC, the panoramic image is sent to the display of the cockpit device through the AVM application on the SoC side;

[0041] The display of the panoramic image on the display of the cockpit device of the vehicle includes:

[0042] When the MCU monitors that the state of the vehicle meets the AVM pop-up condition, a pop-up window signal for popping up the AVM interface is sent to the cockpit device;

[0043] For the cockpit device to display the panoramic image based on the AVM interface after receiving the pop-up window signal.

[0044] According to a panoramic surround cold start image output method provided by the present application, the panoramic image includes images of multiple perspectives,

[0045] The initialization of the image acquisition device and data transmission link of the vehicle includes:

[0046] Initializing the image acquisition device related to the target perspective in the vehicle;

[0047] The receiving and processing of the data of the image acquisition device by the AVM application to generate a panoramic image includes:

[0048] Receiving and processing the data of the image acquisition device by the AVM application to generate a panoramic image of the target perspective.

[0049] According to a panoramic surround cold start image output method provided by the present application, the target perspective is the rear view perspective, the image acquisition device related to the target perspective is the rear view camera, and the panoramic image of the target perspective is the reverse image.

[0050] The present application also provides an intelligent driving controller, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the panoramic surround cold start image output method as described in any one of the above.

[0051] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the panoramic surround cold start image output method as described in any one of the above.

[0052] In the embodiments of this specification, the panoramic surround view cold start image output method, device, and computer-readable storage medium, compared with the long current cold start AVM image output time, by powering on the MCU in the intelligent driving controller to wake up the SoC in the intelligent driving controller, so that the SoC can load the bootloader and boot the startup and initialization of the Linux system kernel. During the startup or initialization of the Linux system kernel, the vehicle's image acquisition device and data transmission link are initialized, advancing the initialization of the camera, and solving the delay problem caused by the initialization after the current service starts. And based on the service configuration file of the Linux system, start the system services in a preset order, and give priority to starting the necessary resources related to AVM image output to ensure that these resources can be called and used in a timely manner during the AVM image output process, avoiding the image output delay caused by resource waiting, thereby saving time. Then, the AVM application receives and processes the data of the image acquisition device, generates a panoramic image, and displays the panoramic image through the display of the vehicle's cockpit device, shortening the time from vehicle cold start to panoramic image display, and realizing fast panoramic surround view cold start image output without increasing hardware costs.

[0053] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit this specification. Brief Description of the Drawings

[0054] The drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments that conform to this specification, and are used together with the specification to explain the principles of this specification.

[0055] Figure 1 is an architecture diagram of a panoramic surround view cold start image output system shown according to an exemplary embodiment of this specification;

[0056] Figure 2 is another architecture diagram of a panoramic surround view cold start image output system shown according to an exemplary embodiment of this specification;

[0057] Figure 3 is a flowchart of a panoramic surround view cold start image output method shown according to an exemplary embodiment of this specification;

[0058] Figure 4 is a timing diagram of the first embodiment of a panoramic surround view cold start image output method shown according to an exemplary embodiment of this specification;

[0059] Figure 5 is a timing diagram of the second embodiment of a panoramic surround view cold start image output method shown according to an exemplary embodiment of this specification;

[0060] Figure 6It is the system architecture diagram of the third embodiment of a panoramic surround cold start image generation method shown in this specification according to an exemplary embodiment;

[0061] Figure 7 It is the timing diagram of the second embodiment of a panoramic surround cold start image generation method shown in this specification according to an exemplary embodiment;

[0062] Figure 8 It is a schematic block diagram of a panoramic surround cold start image generation device shown in this specification according to an exemplary embodiment. Detailed implementation manners

[0063] Here, the technical solutions in the embodiments (or "implementation manners") of the present application will be clearly and completely described in conjunction with the accompanying drawings. When the following description involves the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0064] If there are terms related to directional indications or positional relationships in the embodiments of the present application (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationships and motion conditions between components in a specific posture (as shown in the accompanying drawings); if this specific posture changes, then the directional indication or positional relationship will also change accordingly. In addition, the terms "first", "second", etc. involved in the embodiments of the present application are only for the purpose of convenient description and cannot be understood as indicating or implying relative importance.

[0065] The present application provides a panoramic surround cold start image generation method, device, and computer-readable storage medium. The present application will be described in detail below in conjunction with the accompanying drawings. Without conflict, the features in the following embodiments and implementation manners can be combined with each other.

[0066] In a high-order intelligent driving controller, an SoC (system on chip) and an MCU (Microcontroller Unit) respectively act as a computing unit and a control unit to cooperate to implement the intelligent driving function.

[0067] For vehicles with a high-order intelligent driving platform (such as NVIDIA Orin SoC), there are two ways to meet the AVM fast start requirement.

[0068] One way is to refer to Figure 1The architecture block diagram. An independent controller is added outside the high-order intelligent driving controller to implement the functions of AVM and PAS (Parking Assist System). This independent controller is generally equipped with a low-performance MCU and SoC, and accesses the data of 4 fisheye cameras for bypass transmission to the high-order intelligent driving platform to implement driving intelligent driving functions such as NOA (Navigate on Autopilot).

[0069] In the above method, adding an independent controller will increase the cost, including the costs of the controller, bypass serializer / deserializer chips, and wiring harnesses, etc. Correspondingly, in terms of usage, an additional production line offline, diagnosis, and OTA (Over-The-Air) upgrade system also needs to be developed, which also increases the development and testing costs. In addition, the added controller will also increase the workload of vehicle layout.

[0070] Another method, referring to Figure 2 the architecture block diagram. Connect the fisheye camera / cameras to the DHU (Digital Cockpit Head Unit), and the DHU realizes the function of quickly generating AVM images and bypasses the image data to the intelligent driving platform.

[0071] In the above method, since the failure rate of the current market DHU is relatively high compared to the intelligent driving platform, once a failure causes the AVM function and bypass link to fail, the basic assisted driving function will fail, and the user experience is poor. On the other hand, due to the requirement of the intelligent driving system for the sensor to synchronously trigger exposure, for the surround view cameras connected to the DHU, the implementation of synchronous exposure is relatively complex, and the error of synchronous exposure increases, which may cause the perception effect to deteriorate.

[0072] Therefore, there is an urgent need for a panoramic surround view cold start fast image generation solution for the high-order intelligent driving platform.

[0073] To solve the above technical problems, this specification provides a panoramic surround view cold start image generation method.

[0074] Aiming at not increasing the hardware cost, by optimizing the software / system startup timing, the startup / initialization of each module / service related to AVM image generation is advanced / has its priority increased, and the necessary resources related to AVM image generation, such as specific memory spaces, specific functional modules of the image processing chip, etc., are preferentially started, ensuring that these resources can be called and used in a timely manner during the AVM image generation process, avoiding image generation delays caused by resource waiting, and improving the AVM image generation speed and efficiency.

[0075] Specifically, to address the problem of long camera initialization time during the AVM image generation process, three solutions are proposed, namely initializing the camera in the Linux Kernel, initializing the camera through hardware design of the MCU, and only initializing, processing, and outputting the rear-view fisheye in the first stage.

[0076] Figure 3 FIG. 4 is a schematic flow diagram of a panoramic surround cold start image generation method provided in the first embodiment of the present specification, including the following steps:

[0077] Step S100: Power on the MCU in the intelligent driving controller to wake up the SoC in the intelligent driving controller, so that the SoC loads the bootloader and boots and initializes the kernel of the Linux system;

[0078] Step S200: During the startup or initialization of the kernel of the Linux system, initialize the vehicle's image acquisition device and data transmission link, and configure the data transmission channel between the image acquisition device and the Linux system;

[0079] Step S300: Based on the service configuration file of the Linux system, start system services in a preset order. The system services include the panoramic surround system AVM application and the Camera service;

[0080] Step S400: Receive and process the data of the image acquisition device through the AVM application to generate a panoramic image; the data of the image acquisition device is obtained by the Camera service through the data transmission channel and then sent to the AVM application;

[0081] Step S500: Display the panoramic image through the display of the vehicle's cockpit device.

[0082] It should be noted that the panoramic surround cold start image generation method is applied to the intelligent driving controller, and the intelligent driving controller is applied to the panoramic surround cold start image generation system.

[0083] The panoramic surround cold start image generation system at least includes an image acquisition device, an intelligent driving controller, multiple links of the image acquisition device, and a cockpit device. Among them, each camera link at least includes a group of cameras, and each camera link shares a deserializer and a serializer, and the serializer is connected to the cockpit device.

[0084] The intelligent driving controller described in this specification has the conventional structure and functions of the intelligent driving controller on the current market (for example, the intelligent driving domain controller at least includes a serializer, a deserializer, an SoC module, an MCU module, etc.), which will not be specifically introduced herein.

[0085] The cockpit equipment described in this specification includes at least a DHU (Digital Cockpit Head Unit, the driving information and entertainment host).

[0086] The image acquisition equipment described in this specification includes at least various cameras / fisheye cameras / cameras, etc., which are used to acquire the environmental images around the vehicle.

[0087] The specific steps are as follows:

[0088] In step S100, power on the MCU in the intelligent driving controller to wake up the SoC in the intelligent driving controller, so that the SoC can load the bootloader and guide the startup and initialization of the Linux system kernel.

[0089] As an example, the SoC (System on Chip) is a chip integrating modules such as CPU, GPU, and memory. For example, NVIDIA Orin. Through software startup timing optimization, the AVM image can be output within a predetermined time after cold startup on the NVIDIA Orin platform to meet the relevant requirements.

[0090] Refer to Figure 4 , first, when the vehicle is cold started (such as ignition or power on), the vehicle electronic system starts to supply power, and the high-order intelligent driving controller (based on NVIDIA Orin SoC) and the MCU (microcontroller unit) start synchronously. The startup of the MCU is responsible for the initialization of the vehicle's basic hardware, such as power management and sensor power supply control. At the same time, it sends signals to wake up the SoC, deserializer, and serializer. After the MCU wakes up the SoC, the SoC starts and initializes the CPU, memory, and peripheral interfaces (such as I2C, LVDS).

[0091] Then, load the bootloader. The bootloader is a program that runs before the Linux system kernel. It is responsible for initializing hardware devices, establishing a mapping diagram of the memory space, and preparing for the final call of the operating system kernel. The bootloader is responsible for initializing the hardware, loading the Linux Kernel (the core component of the Linux operating system, which is responsible for managing the system's hardware resources, including the CPU, memory, input / output (I / O) devices, etc., and providing support for the application programs running on the system) into the memory and starting it, and handing over the control right to the kernel to complete the startup of the Linux system. Among them, after the bootloader loads the Linux Kernel into the memory, the Linux Kernel starts to execute, performs operations such as hardware detection, initialization of driver programs, and mounting of the root file system, and gradually establishes a complete operating system running environment.

[0092] In step S200, during the startup or initialization of the Linux system kernel, the image acquisition device and data transmission link of the vehicle are initialized, and the data transmission channel between the image acquisition device and the Linux system is configured.

[0093] The fisheye camera is the image acquisition device of the AVM image output system. During the initialization process, parameters of the camera are set, such as resolution, frame rate, exposure time, etc. At the same time, the data transmission link between the camera sensor and the Linux system is configured to ensure that the camera can normally acquire image data and transmit it to the subsequent processing module.

[0094] As an example, after the SoC starts up, during the startup or initialization of the Linux Kernel, the loading of the fisheye camera driver and the configuration of the LVDS (Low-Voltage Differential Signaling) interface are completed. Specifically, the SoC sends configuration instructions (such as resolution, exposure parameters) to the fisheye camera sensor, serializer, and deserializer through the I2C protocol (Inter-Integrated Circuit, a serial communication protocol), configures the LVDS output format of the serializer and the data parsing mode of the deserializer, and establishes a data transmission channel between the fisheye camera and the Linux system.

[0095] As an example, the data transmission channel is an LVDS transmission link, and the Camera Service sends the data of the image acquisition device to the AVM application in a high-speed and low-noise manner through the LVDS transmission link.

[0096] As an example, the fisheye camera includes at least a sensor and a serializer.

[0097] As an example, the deserializer is used to receive the original camera data and convert it into an LVDS signal for transmission to the SoC.

[0098] As an example, the process of raw image acquisition based on the data transmission channel is as follows: Data is acquired by the fisheye camera and given to the serializer, which converts it into an LVDS signal and then transmits it to the deserializer. The Camera Service (an application service for distributing camera data) receives the deserialized raw image data.

[0099] In the actual test, the fisheye camera sensor and serial deserialization initialization are placed in the Linux Kernel startup phase. This part mainly includes powering the fisheye camera and configuring the fisheye camera sensor and serial deserialization chip using the I2C protocol. Since the initialization of the camera takes 3 to 4 seconds, the Linux Kernel starts about 2.4 seconds after power-on, ensuring that the fisheye camera can be initialized within 6.4 seconds after power-on.

[0100] In this embodiment, the fisheye camera sensor, serializer and deserializer are initialized in the Linux kernel startup phase. SoC control gives priority to initializing the sensor, serializer and deserializer, avoiding the current problem of initializing the fisheye camera after the CameraService is started, which results in a long startup time when there is a demand for AVM output, thereby improving the system response speed and the availability of the AVM function.

[0101] In step S300, based on the service configuration file of the Linux system, the system services are started in a preset order, wherein the system services include the panoramic view system AVM application and the Camera service.

[0102] Reference Figure 4 Systemd is a commonly used system and service manager in Linux systems. Starting systemdservice will start a series of system services according to the configuration file. These services provide various functional supports for the normal operation of the system, such as network services, log services, etc.

[0103] Specifically, systemd will read the system's service configuration files and start each service in sequence according to the dependency and startup order. In the AVM output system, services related to image transmission and processing are started.

[0104] In some embodiments, the system service includes a systemd device trigger service, and the systemd device trigger service includes a device and a subsystem.

[0105] The service configuration file of the Linux system includes:

[0106] Based on the dependency relationship between the Xorg service related to the AVM image output and the device, configure the startup sequence of the subsystem and the device;

[0107] The startup priority of the device is greater than the startup priority of the subsystem.

[0108] The systemd-dev-trigger.service service described in this specification is a sub-service in the System service startup and is closely related to the udev device manager. The main function of this service is to trigger udev rules in response to device events issued by the kernel. The systemd-dev-trigger.service service includes subsystem and device.

[0109] In the systemd-dev-trigger.service service, the startup of the subsystem involves a series of device-related initialization, configuration, and management operations, providing the necessary environment and support for the subsequent normal use of the device and the operation of other services.

[0110] The device described in this specification refers to the hardware devices in a computer system, such as graphics cards, mice, keyboards, cameras, etc. In the Linux system, devices usually have corresponding device files in the / dev directory. Through these files, the operating system can interact and communicate with the hardware devices.

[0111] The Xorg service described in this specification is an important component in the Linux system for providing graphic display services and is an open-source implementation of the X Window System for the graphical user interface. It must be started before the AVM outputs images. That is, after the Xorg service is started, the AVM application transmits the processed panoramic images to the DHU display through the graphic interface of Xorg. And the startup of Xorg depends on the completion of the initialization of specific devices, which means that one of the prerequisite conditions for the startup of the Xorg service is the startup of the device.

[0112] Therefore, modify the systemd-dev-trigger.service service, swap the order of the subsystem and the device, that is, start the device first, advance the device initialization on which Xorg depends, so that Xorg can obtain the required device resources faster, reduce the startup time of the Xorg service, and achieve the fast startup of Xorg.

[0113] As an example, find the configuration file of the systemd-dev-trigger.service service and open the configuration file, find the part related to the startup order of the subsystem and the device and modify it. After the modification is completed, reload the configuration of systemd to make the modification take effect.

[0114] In actual tests, by swapping the order of the subsystem and the device and advancing the initialization of the devices on which Xorg depends, the time taken for the initialization of the Xorg service can be reduced by approximately 0.8 seconds.

[0115] In this embodiment, by adjusting the initialization order of the devices (device) and subsystems (subsystem) in systemd-dev-trigger.service, the device initialization is completed first, thereby triggering the start of the Xorg service in advance and improving the subsequent AVM image rendering efficiency.

[0116] In some embodiments, the system service further includes a mounting service for each partition of the disk, so that the Linux system can access and use the files and data on each partition.

[0117] The service configuration file of the Linux system includes:

[0118] Based on the correlation between each partition of the disk and the AVM application, determine the mounting order for mounting each partition of the disk according to the degree of correlation.

[0119] Among them, the mounting order is that the mounting priorities of multiple partitions that have a direct correlation with the AVM application are higher than those of other partitions of the disk.

[0120] All application software depends on the completion of the mounting of the partition where it is located before it can be loaded and run. The purpose of this measure is to minimize the time taken for the mounting of disk partitions.

[0121] Traditionally, when the system starts, each partition of the disk is mounted sequentially in order. This serial mounting method means that the mounting of each partition needs to wait for the previous partition to be mounted before it can start. When the number of disk partitions is large, for example, there are 20 partitions, the sequential mounting will take a long time, resulting in a slow overall system startup speed. Especially for those application software that need to run as soon as possible after startup and depend on the completion of the mounting of a specific partition, the waiting time will be even longer.

[0122] Based on this, in this specification, the disk partitions related to image rendering are mounted in advance, that is, the disk partitions related to AVM image rendering are mounted first. This can ensure that the partitions related to AVM image rendering can be quickly ready, enabling the AVM image rendering function to start and run earlier, thereby reducing the total time from system startup to AVM image rendering and improving the system response speed and the availability of the AVM function. At the same time, other partitions that are not directly related to AVM image rendering can be mounted slowly later, without affecting the key processes of AVM image rendering.

[0123] Each partition of the disk described in this specification refers to dividing a physical disk into multiple logically independent regions, and each partition can have its own file system and storage purpose.

[0124] Determine the correlation between each partition and the AVM application, and determine the partitions directly related to AVM image generation. For example, a disk partition designed and optimized specifically for the rapid startup of AVM stores the files and data necessary for the startup and image generation of the AVM system. When mounting, it is preferentially mounted to quickly load the key data therein, enabling the AVM to start and generate images quickly.

[0125] When there are multiple disk partitions directly related to AVM image generation, the parallel mounting method is adopted to perform the mounting operations on multiple disk partitions simultaneously. Utilizing the advantages of parallel processing, these partitions are mounted simultaneously, greatly shortening the total time required for mounting. This ensures that the partitions related to AVM image generation can be quickly prepared, enabling the AVM image generation function to start and run earlier, thereby reducing the total time from system startup to AVM image generation.

[0126] Furthermore, reducing the capacity of multiple partitions that have a direct correlation with the AVM application can reduce the mounting time of this partition, enabling the AVM to start and generate images quickly.

[0127] In this embodiment, the disk partitions related to AVM image generation are preferentially and parallelly mounted to reduce the time-consuming of disk partition mounting and shorten the total time from system startup to AVM image generation.

[0128] In some embodiments, the service configuration file based on the Linux system includes:

[0129] After the mounting is completed, immediately start the AVM application and the Camera service.

[0130] The Camera service described in this specification is responsible for managing the fish-eye camera and the image acquisition service. It is used to receive the raw image data sent by the fish-eye camera transmitted by the deserializer and work in cooperation with the AVM application program to provide image data for the AVM application.

[0131] The AVM application described in this specification is the core software for implementing the panoramic view function. It is responsible for receiving the image data collected by the camera, performing image processing and analysis, and generating a panoramic image.

[0132] Continue to refer to Figure 4, after performing the necessary file check and backup operations, immediately call the startup scripts of the Camera service and the AVM application. After the software / functions necessary for AVM image output are started, that is, after the MCU is powered on and the partition is mounted as shown in the following figure, immediately start the Camera service and the AVM application.

[0133] The file check and backup operations described in this specification are to ensure the integrity and availability of key files in the system. In the AVM system, back up and repair configuration files, image data, etc. related to the AVM application to prevent system failures caused by data loss or corruption, ensure the security and reliability of system data, and avoid affecting the normal operation of the AVM system due to data problems.

[0134] In this embodiment, the Camera service and the AVM application are started early to ensure that the AVM application can immediately obtain the image data collected by the fisheye camera after startup, improving the image output speed and efficiency of the AVM.

[0135] In some embodiments, the system service includes a data distribution service.

[0136] The service configuration file based on the Linux system includes:

[0137] The Camera service distributes the data of the image acquisition device to the AVM application through the dynamic multicast mechanism.

[0138] As an example, the specific implementation of the dynamic multicast mechanism early consumer described in this specification includes:

[0139] When the Camera Service starts, create a dynamic multicast group and open a registration channel for image consumers to access; the AVM application registers as the primary consumer to the multicast group, triggering the Camera Service to perform data transmission; other image consumers perform asynchronous registration after the AVM application has completed registration and dynamically join the multicast group to receive the data stream.

[0140] In multicast communication, there is usually one or more image producers and multiple image consumers. Among them, the Camera service is the image producer, the AVM application refers to the image consumer, and data acquisition, E2E functions, etc. are other image consumers.

[0141] Traditional static multicast (a static image distribution component of NV DriveOS) requires the CameraService to establish communication connections with all image consumers (including AVM applications, data collection, E2E functions, and other auxiliary function modules) first. After completing all the preparatory work, it will start distributing images to each consumer. This means that it is necessary to wait for all consumers to be ready. Even if there is a problem or a delay in connecting to one of the consumers, it will cause a delay in the entire image distribution process.

[0142] Therefore, when using dynamic multicast (a dynamic image distribution component of NV DriveOS) and introducing the early consumer mechanism, in the case of multiple image consumers, the Camera Service can distribute image data to all registered consumers in parallel through the dynamic multicast mechanism. When the AVM application is used as an image consumer, the image transmission between the Camera service and the AVM application is achieved through the early consumer mechanism.

[0143] Specifically, after the AVM application starts, it will actively establish a connection with the Camera service and mark the AVM application as an early consumer. When the Camera Service starts as an image producer, once it establishes a connection with the designated early consumer, that is, the AVM application, it immediately starts transmitting image data to it without having to wait for all other image consumers to complete the connection and preparatory work, thereby reducing the image transmission link establishment time. Other consumers can gradually complete the connection and preparatory work in the subsequent process without affecting the AVM application's timely acquisition of image data.

[0144] This mechanism takes advantage of the flexibility of multicast communication. By identifying the key consumers in advance and providing services to them preferentially, it reduces the overall waiting time, thereby improving the efficiency of image transmission and reducing the waiting time for the AVM to display images. In actual tests, using the early consumer mechanism in dynamic multicast can reduce the time consumption by approximately 0.6 seconds compared to static multicast.

[0145] Through the above embodiments, measures such as partition mounting, Xorg service optimization, and adjustment of the application startup sequence ensure that the AVM application starts at the fastest speed; the early consumer mechanism of dynamic multicast is used to ensure the establishment of the image transmission link at the fastest speed; and through the optimization of the interaction logic between the MCU and the SOC in the subsequent chapters, it is ensured that the image output no longer depends on the startup of the communication module. Therefore, the time for the AVM to output an image is reduced to varying degrees in the provided system services, and the efficiency of the AVM cold start to output an image is improved.

[0146] In step S400, the AVM application receives and processes the data of the image acquisition device to generate a panoramic image; the data of the image acquisition device is obtained by the Camera service through the data transmission channel and then sent to the AVM application.

[0147] The SoC starts the Camera Service and distributes the original images to the AVM application through the dynamic multicast mechanism. The AVM application receives the data in real time and performs relevant processing to obtain a panoramic image. The relevant processing at least includes distortion correction, stitching, rendering, fusion, etc. to synthesize the images collected by multiple cameras into a complete panoramic image.

[0148] The SoC pushes the processed panoramic image video stream to the DHU through the LVDS interface. The DHU continuously receives the video stream and displays the panoramic image through the AVM interface. However, in the traditional image output interaction logic, whether the AVM interface pops up is controlled by the MCU. This is because the signal requesting the DHU to pop up the AVM interface is sent by the MCU. Therefore, a communication interaction process is required between the SoC and the MCU to ensure that the MCU requests image output only after the AVM application image on the SoC side is ready.

[0149] The specific image output interaction logic is as follows: First, the MCU controls the SoC to produce and output the video stream (the SoC will only output the video stream to the DHU when it receives the signal from the MCU. However, at this time, the DHU has no window to display. Only when the MCU controls the DHU display window can the image in the window be displayed by the DHU). Then, the MCU controls the DHU display window, and the video stream is displayed in the window.

[0150] The above interaction logic means that a complete communication connection needs to be established between the MCU and the SoC during image output. Since the startup of the communication component is slow and takes a long time, it causes image output delay and affects the user experience, especially when image information is urgently needed after the vehicle starts, such as the timely response of functions like reverse image.

[0151] Therefore, in view of the above problems, the interaction logic between the MCU and the SoC is improved. The SoC produces and outputs the video stream without relying on the control of the MCU in the initial stage of vehicle startup. Instead, within the time threshold range, the SoC independently outputs and produces the video stream. Reducing the time waiting for the communication connection between the MCU and the SoC can display the image on the DHU faster, thereby improving the response speed of the system.

[0152] As an example, the method further includes:

[0153] Within the set time range when the SoC starts up, the panoramic image is sent to the display of the cockpit device through the AVM application at the SoC end;

[0154] The display of the panoramic image through the display of the vehicle's cockpit device includes:

[0155] When the MCU monitors that the state of the vehicle meets the AVM pop-up condition, a pop-up window signal for popping up the AVM interface is sent to the cockpit device;

[0156] So that after the cockpit device receives the pop-up window signal, the panoramic image is displayed based on the AVM interface.

[0157] The set time range described in this specification can cover the time from vehicle startup to the start of these operations, ensuring that when the driver needs to use the AVM function, the system can respond in a timely manner and provide effective image information for assisted driving. Considering that the SOC needs a certain amount of time to initialize various hardware resources and software modules after startup, and at the same time, it also needs to process image data. A relatively reasonable time range is set comprehensively, which can not only ensure that the SOC has enough time to prepare and output image data, but also meet the real-time requirements of the overall system. As an example, the set time range is 20 seconds, and 20 seconds of time can usually cover the time from vehicle startup to the start of these operations.

[0158] In the optimized logic, the SoC directly outputs the video stream within the set time range after startup, without waiting for the request signal of the MCU, ensuring that the picture is ready in real time and avoiding the problem of slow startup of the communication component.

[0159] The MCU does not need to rely on the interaction with the SoC. The MCU only needs to send a pop-up window signal for popping up the AVM interface to the DHU when the AVM pop-up condition is met, without relying on the communication confirmation of the SoC. Among them, the MCU monitors the vehicle state and judges whether the vehicle state meets the AVM pop-up condition. The vehicle state includes but is not limited to shifting into reverse gear, the user clicking the AVM button, the reverse signal, etc.

[0160] It should be noted that outside the set time range of the SoC startup, the system usually switches back to the above-mentioned traditional image output interaction logic to ensure the output of high-quality AVM images.

[0161] In this embodiment, the pop-up window signal of the MCU is decoupled from the video stream output of the SOC, avoiding the problem of slow startup of the communication module between the MCU and the SoC, and ensuring that the DHU immediately displays the image. Since the SOC has output the video stream in advance, the display delay of the DHU only depends on the signal reception time. Even if the SOC-MCU communication is abnormal, the DHU can still receive and cache the video stream and immediately display it after the pop-up window signal arrives, reducing the cold startup delay and improving the fault tolerance rate.

[0162] In step S500, the panoramic image is displayed on the display of the vehicle's cockpit device.

[0163] After the DHU receives the pop-up window signal, it immediately displays the panoramic image interface transmitted by the SOC.

[0164] Through the above embodiments, referring to Figure 4 , in actual testing or use, on the high-level intelligent driving platform for integrated parking and driving, the optimization measures for the controller startup timing can ultimately achieve the display of the AVM picture on the DHU within 7.8 seconds from the controller cold startup without increasing the hardware and wiring harness costs, providing a better vehicle use experience for users. For safety reasons, there are clear regulatory restrictions in some regions on the time from vehicle cold startup to AVM image output (for example, FMVSS-111 in North America requires that the reverse image must be displayed within 6 seconds after shifting to the R gear within 2 seconds of vehicle cold startup, and EU R158e requires that the reverse image must be displayed within 8 seconds after shifting gears during vehicle cold startup). For vehicles that do not meet the regulations and cannot enter the corresponding market for sales, the solution in this embodiment optimizes the process from vehicle cold startup to AVM image output, and realizes that the time to complete AVM image output on the Orin platform meets the corresponding regulations, enabling the vehicle to be legally sold in the corresponding regions.

[0165] It should be noted that this solution is proposed for the intelligent driving platform. When the AVM function is deployed on the cockpit side, the cockpit side can also implement the rapid output of AVM based on these measures of this solution.

[0166] The present application provides a panoramic surround view cold start image output method, device, and computer-readable storage medium. Compared with the long cold start AVM image output time currently, by powering on the MCU in the intelligent driving controller to wake up the SoC in the intelligent driving controller, the SoC is provided to load the bootloader and guide the startup and initialization of the Linux system kernel. During the startup or initialization of the Linux system kernel, the vehicle's image acquisition device and data transmission link are initialized, advancing the initialization of the camera, and solving the delay problem caused by the initialization after the current service starts. And based on the service configuration file of the Linux system, the system services are started in a preset order, and the necessary resources related to AVM image output are preferentially started to ensure that these resources can be called and used in a timely manner during the AVM image output process, avoiding the image output delay caused by resource waiting, thereby saving time. Then, the AVM application receives and processes the data of the image acquisition device, generates a panoramic image, and displays the panoramic image through the display of the vehicle's cockpit device, shortening the time from vehicle cold start to panoramic image display, and realizing fast panoramic surround view cold start image output without increasing the hardware cost.

[0167] Based on the above first embodiment, a second embodiment of the panoramic surround view cold start image output method is proposed.

[0168] In the traditional solution, initializing and configuring 4 cameras usually takes 3 to 4 seconds. Since the regulations in relevant regions only stipulate the output time of a certain perspective image during cold start, based on the image output processes in the first embodiment, the fisheye camera for this perspective is initialized and configured to achieve the effect of preferentially outputting the panoramic image of this perspective.

[0169] In some embodiments, the panoramic image includes images of multiple perspectives.

[0170] The initialization of the vehicle's image acquisition device and data transmission link includes:

[0171] Initializing the image acquisition device related to the target perspective in the vehicle;

[0172] The process of receiving and processing the data of the image acquisition device by the AVM application to generate a panoramic image includes:

[0173] Receiving and processing the data of the image acquisition device by the AVM application to generate the panoramic image of the target perspective.

[0174] Generally, there are four fisheye cameras in the front, rear, left, and right of the vehicle, which can display panoramic images of multiple perspectives. Therefore, when displaying the panoramic image of the target perspective through the DHU, the image acquisition device related to the target perspective can be initialized and configured specifically.

[0175] Compared with the traditional solution where initializing and configuring 4 cameras usually takes 3 to 4 seconds, the solution corresponding to the second embodiment focuses on a single fisheye camera. With a simplified operation process, the configuration time is significantly reduced to about 1.5 seconds. This means that during system cold start, the preparation of the fisheye camera for the key perspective can be quickly completed, and the output of the corresponding image can be achieved preferentially.

[0176] As an example, the target perspective is the rear view perspective, the image acquisition device related to the target perspective is the rear view camera, and the panoramic image of the target perspective is the reverse image.

[0177] Since the regulations in Europe and America only stipulate the output time of the reverse image during cold start, taking the rear view perspective as the target perspective, during the initialization process of the fisheye camera in the first embodiment, only the rear view fisheye camera / rear view camera is initialized and configured, so that the preparation of the key rear view fisheye camera can be quickly completed, and the output of the reverse image can be achieved preferentially. This solution not only meets the requirements of European and American regulations, but also significantly improves the output efficiency of the reverse image through process optimization, providing timely and necessary visual assistance to the driver at the initial stage of vehicle startup and enhancing driving safety.

[0178] In other embodiments, refer to Figure 5 and design the rear view camera to be initialized after the Camera Service starts to achieve the effect of preferentially outputting the reverse image.

[0179] Through the above embodiments, for specific scenario optimization, focusing on the initialization of key cameras, time is saved, and rapid image output during panoramic surround view cold start is achieved.

[0180] Based on the above first embodiment or second embodiment, a third embodiment of the method for panoramic surround view cold start image output is proposed.

[0181] In the traditional architecture, the initialization and control of the fisheye camera sensor and the serial deserializer usually rely on the SoC. However, the startup process of the SoC is relatively complex and time-consuming, which causes the initialization of the fisheye camera to lag, thus affecting the overall image output efficiency of the AVM.

[0182] Refer to Figure 6 's architecture diagram. The purpose of this embodiment is to design a solution in hardware to solve the problem that the late initialization of the fisheye camera leads to a long image output time of the AVM.

[0183] That is, in the first embodiment, the initialization and control of the fisheye camera are operated by the SoC. In this embodiment, the initialization of the fisheye camera and the deserializer is moved to start after the MCU starts.

[0184] As an example, the panoramic surround view cold start image generation method further includes:

[0185] Establish a communication connection between the MCU and the deserializers in the intelligent driving controller, where the deserializers are connected to the SoC;

[0186] After the MCU is started, initialize the vehicle's image acquisition device and data transmission link based on the MCU.

[0187] Refer to Figure 7 , establish a communication connection between the MCU and the deserializers. The MCU can efficiently send initialization instructions to the fish-eye camera sensor and accurately set its working parameters. At the same time, initialize the serializer and deserializers to ensure the smoothness of the data transmission link. Since the startup time of the MCU (generally within 2 seconds) is much shorter than that of the SoC, the initialization configuration of the camera can be synchronized by the MCU during the startup process of the SoC. It should be noted that the image processing application still runs on the SoC.

[0188] In this embodiment, based on the software optimization of Embodiment 1, hardware division of labor is introduced. By directly controlling the initialization of the fish-eye camera sensor, serializer, and deserializers through the MCU, an effective way is provided to shorten the AVM image generation time and improve the overall system performance.

[0189] Figure 8 Illustrates a schematic diagram of the physical structure of a panoramic surround view cold start image generation device. As shown in Figure 8 , the panoramic surround view cold start image generation device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the panoramic surround view cold start image generation method.

[0190] In addition, when the logical instructions in the above-mentioned memory 830 can be implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, external hard drives, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0191] On the other hand, this application also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the panoramic surround cold start image generation method provided by the above-mentioned various methods.

[0192] On another aspect, this application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the panoramic surround cold start image generation method provided by the above-mentioned various methods.

[0193] It should be noted that the technical solutions or technical features described in the above embodiments can be combined or supplemented with each other without conflict. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the drawings; all modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this application shall be included within the scope of protection of this application.

Claims

1. A panoramic surround cold start image generation method, characterized in that Applied to an intelligent driving controller, the method includes: Power on the MCU in the intelligent driving controller to wake up the SoC in the intelligent driving controller, so that the SoC loads the bootloader and boots and initializes the kernel of the Linux system; During the startup or initialization of the kernel of the Linux system, initialize the vehicle's image acquisition device and data transmission link, and configure the data transmission channel between the image acquisition device and the Linux system; Based on the service configuration file of the Linux system, start system services in a preset order, and the system services include the Around View Monitor (AVM) application and the Camera service; Receive and process the data of the image acquisition device through the AVM application to generate a panoramic image; the data of the image acquisition device is obtained by the Camera service through the data transmission channel and sent to the AVM application; Display the panoramic image through the display of the vehicle's cockpit device.

2. The method for generating an image during cold start of the Around View Monitor as claimed in claim 1, wherein: The method further includes: Establish a communication connection between the MCU and the deserializer in the intelligent driving controller, and the deserializer is connected to the SoC; After the MCU starts, initialize the vehicle's image acquisition device and data transmission link based on the MCU.

3. The method for generating an image during cold start of the Around View Monitor as claimed in claim 1, wherein: The system service includes the systemd device trigger service, and the systemd device trigger service includes a subsystem and a device, The service configuration file of the Linux system includes: Configure the startup order of the subsystem and the device based on the dependency relationship between the Xorg service related to AVM image generation and the device; Wherein, the startup priority of the device is higher than that of the subsystem.

4. The method for generating an image during cold start of the Around View Monitor as claimed in claim 1, wherein: The system service further includes a mounting service for each partition of the disk, so that the Linux system can access and use the files and data on each partition, The service configuration file of the Linux system includes: Based on the correlation between each partition of the disk and the AVM application, determine the mounting order for mounting each partition of the disk according to the degree of correlation; Wherein, the mounting order is that the mounting priorities of multiple partitions having a direct correlation with the AVM application are higher than other partitions of the disk.

5. The method for generating an image during cold start of the Around View Monitor as claimed in claim 4, wherein: Multiple partitions having a direct correlation with the AVM application are mounted in parallel; And / or Reduce the capacity of multiple partitions having a direct correlation with the AVM application.

6. The method for generating an image during cold start of the Around View Monitor as claimed in claim 5, wherein: The service configuration file based on the Linux system includes: After the mounting is completed, the AVM application and the Camera service are started immediately.

7. The panoramic cold-start image output method according to claim 1, characterized in that: The system services include data distribution services, The service configuration file based on the Linux system includes: The Camera service distributes the data of the image acquisition device to the AVM application through a dynamic multicast mechanism.

8. The panoramic cold-start image output method according to claim 7, characterized in that: The dynamic multicast mechanism includes an image consumer and an image producer. The image consumer includes the AVM application and other auxiliary function modules, and the image producer is the Camera service. The method further comprises: In the case of multiple image consumers, the Camera Service distributes image data to all registered consumers in parallel through a dynamic multicast mechanism.

9. The panoramic cold start image output method according to claim 8, characterized in that: The dynamic multicast mechanism includes: When the Camera Service starts, a dynamic multicast group is created and a registration channel is opened for image consumers to access; The AVM application registers to the multicast group as the primary consumer and triggers the Camera Service to transmit data; Other consumers perform asynchronous registration after the AVM application registration is completed and dynamically join the multicast group to receive data streams.

10. The panoramic cold start image output method according to claim 1, characterized in that: The method further comprises: Sending the panoramic image to the display of the cockpit device through the AVM application on the SoC end within a set time range when the SoC is started; The display of the panoramic image on a display of a cockpit device of the vehicle includes: When the MCU detects that the state of the vehicle meets the AVM pop-up condition, it sends a pop-up signal for popping up the AVM interface to the cockpit device; So that after the cockpit device receives the pop-up signal, it can display the panoramic image based on the AVM interface.

11. The panoramic cold start image output method according to claim 1, characterized in that: The panoramic image includes images from multiple perspectives. Initializing the image acquisition device and the data transmission link of the vehicle includes: Initializing an image acquisition device in the vehicle that is related to a target viewing angle; The step of receiving and processing the data of the image acquisition device through the AVM application to generate a panoramic image includes: The AVM application receives and processes data from the image acquisition device to generate a panoramic image of the target viewing angle.

12. The panoramic cold start image output method according to claim 11, characterized in that: The target viewing angle is a rear-view viewing angle, the image acquisition device related to the target viewing angle is a rear-view camera, and the panoramic image of the target viewing angle is a reversing image.

13. An intelligent driving controller, characterized in that, It includes a memory, a processor, and a panoramic surround cold start mapping program stored on the memory and executable on the processor. When the processor executes the panoramic surround cold start mapping program, it implements the steps of the panoramic surround cold start mapping method according to any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, A panoramic surround cold start mapping program is stored on the computer-readable storage medium. When the panoramic surround cold start mapping program is executed, it implements the steps of the panoramic surround cold start mapping method according to any one of claims 1-12.

Images