Method for generating panoramic surround view image, intelligent driving system and vehicle

Abstract

The application provides a panoramic surround view image generation method, an intelligent driving system and a vehicle. In the case that a first controller completes power-on starting and a second controller does not complete power-on starting, the first controller respectively sends a first enabling command and a collection control command to a transcoding channel unit and an image collection unit; the image collection unit collects a video stream based on the collection control command and transmits the video stream to the transcoding channel unit; and the transcoding channel unit switches to a working mode based on the first enabling command, converts the format of the video stream and transmits the video stream to a cabin controller to generate a panoramic surround view image. In the power-on starting stage of the second controller, the first controller controls the image collection unit to collect a video stream and transmits the video stream to the cabin controller through the transcoding channel unit, so that the cabin controller can quickly generate a panoramic surround view image based on the received video stream, and a user can quickly use the AVM function without waiting for the second controller to complete starting in the cold starting stage.

Description

Technical Field

[0001] This invention relates to the field of driver assistance technology, and more specifically, to a method for generating panoramic surround view images, an intelligent driving system, and a vehicle. Background Technology

[0002] Currently, cars equipped with intelligent driving systems are generally equipped with AVM (Around View Monitor) functionality. This function uses multiple (usually four) ultra-wide-angle fisheye lenses installed around the vehicle to collect images, and then uses image processing algorithms to correct distortion and stitch the images together to generate a panoramic image of the vehicle's surrounding environment. AVM functionality can effectively eliminate blind spots and provide drivers with safer and more convenient environmental perception support during parking and driving.

[0003] In existing technologies, during the cold start phase of a car, the AVM function can only be enabled to output a panoramic surround view image after the SoC (System on Chip) has completed its startup. However, the startup time of the SoC is generally long (e.g., more than ten seconds), which makes users wait a long time to see the panoramic surround view image when the vehicle is cold started. Summary of the Invention

[0004] The purpose of this invention is to provide a method for generating panoramic surround view images, an intelligent driving system, and a vehicle, so as to improve the problems existing in the prior art.

[0005] The embodiments of the present invention can be implemented as follows: In a first aspect, the present invention provides a method for generating a panoramic surround view image, the method comprising: When the first controller completes power-on startup but the second controller does not complete power-on startup, the first controller sends a first enable command and an acquisition control command to the transcoding channel unit and the image acquisition unit, respectively. The image acquisition unit acquires a video stream representing the external environment of the vehicle based on the acquisition control command, and transmits it to the transcoding channel unit; After the transcoding channel unit switches to working mode based on the first enable command, it performs format conversion on the received video stream and transmits the converted video stream to the cockpit controller. The cockpit domain controller generates a panoramic surround view image based on the video stream.

[0006] In an optional implementation, before the image acquisition unit acquires a video stream characterizing the external environment based on the acquisition control command, the method further includes: When the second controller completes power-on startup, the second controller sends the first enable command and the acquisition control command to the transcoding channel unit and the image acquisition unit, respectively.

[0007] In an optional implementation, the method further includes: When the second controller completes the power-on startup, the second controller sends a startup success signal to the first controller; When the first controller receives the start-up success signal, it sends a control switch signal to the cockpit domain controller. Upon receiving the control switch signal, the cockpit domain controller generates the panoramic surround view image based on the vehicle status and at least the last video frame of the video stream.

[0008] In an optional implementation, the step of the cockpit domain controller generating the panoramic surround view image based on the vehicle status and at least the last video frame of the video stream after receiving the control switch signal includes: If the car is stationary, the cockpit domain controller generates a panoramic surround view image based on the last video frame until the video stream is received again. If the vehicle is in a low-speed motion state, the cockpit domain controller continuously generates a panoramic surround view image based on the last N video frames of the video stream using an optical flow algorithm until the video stream is received again.

[0009] In an optional implementation, after the second controller sends the first enable command and the acquisition control command to the transcoding channel unit and the image acquisition unit respectively, the method further includes: When the cockpit domain controller receives the monitoring start signal and the parking signal, it sends a low-power monitoring signal to the first controller. When the first controller receives the low-power monitoring signal, it controls the second controller to go into sleep mode. The first controller sends a first enable command and an acquisition control command to the transcoding channel unit and the image acquisition unit, respectively.

[0010] In an optional implementation, before the first controller sends the first enable command and the acquisition control command to the transcoding channel unit and the image acquisition unit respectively, the method further includes: The first controller configures the transcoding channel unit to start the transcoding channel unit; The first controller saves the corresponding configuration information when the startup configuration is successful.

[0011] In an optional implementation, before the second controller sends the first enable command and the acquisition control command to the transcoding channel unit and the image acquisition unit respectively, the method further includes: The second controller controls the transcoding channel unit to perform a configuration reset; The second controller performs takeover configuration on the transcoding channel unit and saves the corresponding configuration information when the takeover configuration is successful.

[0012] In an optional implementation, the transcoding channel unit includes a first transcoder and a second transcoder; the startup configuration includes: The first controller configures the first transcoder and the second transcoder respectively; The takeover configuration includes: The second controller sends a second enable command to the first transcoder and the second transcoder, so that both the first transcoder and the second transcoder switch to low power mode based on the second enable command; The second controller configures the first transcoder and the second transcoder respectively.

[0013] Secondly, the present invention provides an intelligent driving system, the intelligent driving system comprising an intelligent driving domain controller, a cockpit domain controller, and an image acquisition unit, wherein the intelligent driving domain controller comprises a first controller, a second controller, and a transcoding channel unit; the first controller is connected to the second controller, the image acquisition unit, and the transcoding channel unit; the second controller is connected to the image acquisition unit and the transcoding channel unit; and the transcoding channel unit is also connected to the cockpit domain controller; wherein: When the first controller completes power-on startup but the second controller does not complete power-on startup, the first controller is used to send a first enable command and an acquisition control command to the transcoding channel unit and the image acquisition unit, respectively. The image acquisition unit is used to acquire a video stream representing the external environment of the vehicle based on the acquisition control command, and transmit it to the transcoding channel unit; The transcoding channel unit is used to switch to the working mode based on the first enable command, convert the format of the received video stream, and transmit the converted video stream to the cockpit controller. The cockpit domain controller is used to generate a panoramic surround view image based on the video stream.

[0014] Thirdly, the present invention provides a vehicle including the intelligent driving system as described in the second aspect above.

[0015] Compared with existing technologies, embodiments of the present invention provide a method for generating a panoramic surround view image, an intelligent driving system, and a vehicle. The method involves: when the first controller has completed its power-on startup but the second controller has not, the first controller sends a first enable command and an acquisition control command to the transcoding channel unit and the image acquisition unit, respectively; the image acquisition unit, based on the acquisition control command, acquires a video stream representing the external environment of the vehicle and transmits it to the transcoding channel unit; the transcoding channel unit, after switching to its working mode based on the first enable command, performs format conversion on the received video stream and transmits the converted video stream to the cockpit controller; the cockpit domain controller generates a panoramic surround view image based on the video stream. In the power-on startup phase of the second controller, the first controller is responsible for controlling and implementing the AVM function. This directly establishes a data channel from the image acquisition unit, through the transcoding channel unit, to the cockpit domain controller, allowing the cockpit domain controller to quickly generate a panoramic surround view image based on the received video stream. During cold starts, there is no need to wait for the second controller to complete startup, and users can quickly use the AVM function. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is one of the architecture diagrams of an intelligent driving system provided in an embodiment of the present invention.

[0018] Figure 2 This is one of the flowcharts illustrating a method for generating a panoramic view image according to an embodiment of the present invention.

[0019] Figure 3 This is a second architecture diagram of an intelligent driving system provided in an embodiment of the present invention.

[0020] Figure 4 The flowchart illustrates the interaction process of the first controller leading the implementation of the AVM function in an embodiment of the present invention.

[0021] Figure 5 The following is an interactive flowchart illustrating the second controller's implementation of the AVM function in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0025] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0026] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0027] Here, we will first introduce the keywords or key terms involved in this invention: 1. ADC (Autonomous Driving Controller): The intelligent driving domain controller (which can be abbreviated as intelligent driving domain controller) is the "autonomous driving brain" of intelligent vehicles. It is mainly responsible for processing data from sensors such as cameras and lidar, performing environmental perception, decision-making and planning, and vehicle control to realize assisted driving functions such as highway navigation and automatic parking.

[0028] 2. CDC (Cockpit Domain Controller): The cockpit domain controller (also known as the vehicle infotainment domain controller) is the "interaction hub" of a smart car. It is responsible for driving multiple screens such as the in-vehicle central control screen, instrument panel screen, and HUD (Head-Up Display) system, and running applications such as voice assistant, map navigation, and audio-visual entertainment, providing drivers and passengers with an intuitive and smooth human-computer interaction experience.

[0029] 3. I2C (Inter-Integrated Circuit) bus: also known as integrated circuit interconnect bus, is a synchronous, multi-master, multi-slave, serial communication bus used to connect microcontrollers and their peripheral devices.

[0030] 4. PMIC: Power Management Integrated Circuit; in an ADC, it is used to supply power to other parts of the ADC under the control of the MCU.

[0031] 5. Sentry Mode: After the vehicle is turned off and locked, a monitoring system consisting of high-definition cameras, ultrasonic radar, inertial acceleration sensors, and high-performance domain controllers remains continuously activated to perform multimodal perception and data analysis of the vehicle's surrounding environment. Once the system identifies a potential threat (such as abnormal approach of people, vehicle collision, glass breakage, etc.) through its built-in algorithms, it will trigger a local alarm (such as horn and flashing lights) and send a real-time notification and on-site video recording to the owner's mobile terminal via the network.

[0032] This invention provides a method for generating panoramic surround view images. During the SoC power-on startup phase, the MCU is responsible for controlling the acquisition of the video stream, which is then transmitted to the CDC through an intermediate transcoding channel unit, thereby reducing the waiting time for users to use the AVM function during cold starts.

[0033] Here, we will first introduce the application scenarios provided by the embodiments of the present invention.

[0034] Please see Figure 1 , Figure 1 The intelligent driving system shown includes an intelligent driving domain controller, a cockpit domain controller, and an image acquisition unit. The intelligent driving domain controller includes a first controller, a second controller, and a transcoding channel unit.

[0035] The first controller is connected to the second controller, the image acquisition unit, and the transcoding channel unit. The second controller is connected to both the image acquisition unit and the transcoding channel unit. The transcoding channel unit is also connected to the cockpit domain controller.

[0036] The first controller can be an MCU (Microcontroller Unit), and the second controller can be a SoC (System-on-a-Chip).

[0037] Combination Figure 1 The intelligent driving system shown in the figure will be followed by a detailed description of the method for generating panoramic surround view images provided by the present invention.

[0038] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a method for generating a panoramic surround view image according to an embodiment of the present invention. The subject executing this method can be the aforementioned intelligent driving system. The method for generating a panoramic surround view image includes the following steps S101 to S104.

[0039] S101, when the first controller has completed power-on startup but the second controller has not completed power-on startup, the first controller sends a first enable command and an acquisition control command to the transcoding channel unit and the image acquisition unit, respectively.

[0040] S102, the image acquisition unit acquires video streams representing the external environment of the vehicle based on acquisition control commands and transmits them to the transcoding channel unit.

[0041] S103, after the transcoding channel unit switches to working mode based on the first enable command, it performs format conversion on the received video stream and transmits the converted video stream to the cockpit controller.

[0042] S104, the cockpit domain controller generates a panoramic surround view image based on the video stream.

[0043] The panoramic surround view image generation method provided in this embodiment of the invention involves the first controller completing the power-on startup when the car is cold-started, followed by the second controller entering the power-on startup phase. Since the startup time of the second controller is relatively long, during the power-on startup process of the second controller, the first controller first controls the image acquisition unit to acquire the video stream and transmits it to the cockpit domain controller through the transcoding channel unit. The cockpit domain controller can obtain the video stream to generate the panoramic surround view image without waiting for the second controller to complete startup, thereby reducing the waiting time for users to use the AVM function during cold starts.

[0044] In an optional implementation, the intelligent driving domain controller also includes a PMIC. The first controller first completes the power-on startup, and then immediately notifies the PMIC to power on the second controller, the transcoding channel unit and the image acquisition unit. In this way, the second controller starts up as soon as it is powered on.

[0045] It is understandable that when the transcoding channel unit is first powered on, it is in low-power mode by default and its internal configuration is set to default values, making it unable to transmit data normally. Therefore, in order to ensure that the transcoding channel unit can transmit data normally, the first controller needs to match the transcoding channel unit first, that is, the method also includes step S100.

[0046] S100, the first controller performs startup configuration on the transcoding channel unit to start the transcoding channel unit, and saves the corresponding configuration information when the startup configuration is successful.

[0047] In this embodiment, after the first controller completes power-on startup and controls the second controller, transcoding channel unit, and image acquisition unit to power on, it will execute steps S100, S101~S104 sequentially. In this way, after the transcoding channel unit is configured to start up, it is controlled to switch to the working mode, so that the video stream transmitted by the subsequent image acquisition unit can be correctly converted in format and transmitted to the cockpit domain controller.

[0048] In the prior art, a deserializer is connected between the second controller and the image acquisition unit. The AVM function is generally controlled by the second controller of the intelligent driving system. Specifically, when the second controller completes power-on, it controls the image acquisition unit to acquire a video stream. The video stream is directly transmitted to the second controller through the intermediate deserializer. Then, the second controller transmits the video stream to the cockpit domain controller, which then continuously generates panoramic surround view images based on the video stream.

[0049] However, during the cold start phase, the first controller typically powers on first (usually taking milliseconds), and then controls the second controller to power on. Because the second controller needs to run numerous application loads, its startup time is generally long (sometimes up to ten seconds). Therefore, the current technology's method of having the second controller dominate the AVM function means that users need to wait a considerable amount of time before they can use the AVM function to see the panoramic surround view image during a cold start.

[0050] Based on existing technology, if the first controller directly takes the lead in controlling the AVM function during the second controller's startup phase, the video stream output by the deserializer cannot be transmitted to the cockpit domain controller because the SoC is starting up, resulting in video stream interruption. Furthermore, even if the deserializer is directly connected to the cockpit domain controller, the video stream output by the deserializer is in MIPI CSI-2 (Mobile Industry Processor Interface Camera Serial Interface Version 2) format. However, the cockpit domain controller normally decodes GMSL (Gigabit Multimedia Serial Link) format and cannot directly decode MIPI CSI-2 format.

[0051] Therefore, to ensure that the video stream can be stably transmitted to the cockpit domain controller and correctly decoded during the startup phase of the second controller, the transcoding channel unit of this invention may include a first transcoder (i.e., a deserializer) and an additional second transcoder (i.e., a serializer), such as... Figure 3 The PMIC is connected to the first controller, the second controller, the image acquisition unit, the first transcoder, and the second transcoder. The first transcoder is connected to the first controller, the second controller, the second transcoder, and the image acquisition unit. The second transcoder is connected to the first controller, the second controller, and the cockpit domain controller. In this way, the second transcoder can replace the SoC in converting MIPI CSI-2 format video streams to GMSL (Gigabit Multimedia Serial Link) format and then directly transmitting them to the cockpit domain controller.

[0052] Optionally, the configuration process involves the first controller configuring the first transcoder and the second transcoder respectively. Next, we will combine... Figure 3 and Figure 4 The process of the first controller leading the implementation of AVM functionality is described.

[0053] The first controller can obtain a stored preset initialization configuration script, which includes configuration information for the first transcoder and the second transcoder. The first transcoder and the second transcoder can be a deserializer and a serializer, respectively, combined... Figure 3 Based on the initialization configuration script, the configuration process for the first controller can be as follows: First, configure the parameters and frame synchronization of the first transcoder via the I2C bus, and configure the output format of the first transcoder to MIPI CSI-2 format; then configure the parameters and link configuration information of the second transcoder via the I2C bus, and configure the output format of the second transcoder to GMSL format; finally, save the initialization configuration script to its own local storage.

[0054] After configuration, the first controller then sends a first enable command to the first transcoder and the second transcoder. The first transcoder switches its output enable pin to the working mode, namely HS (High-Speed) mode, based on the first enable command. The second transcoder switches its input enable pin to the working mode based on the first enable command.

[0055] The first controller can then send an acquisition control command to the image acquisition unit. This acquisition control command includes a continuously sent exposure enable signal, which is a hardware control signal used to synchronously trigger the surround-view camera (Camera Sensor) of the image acquisition unit to acquire images. It is usually represented by a precisely timed level pulse.

[0056] Next, the image acquisition unit continues to acquire images based on the received exposure enable signal. Since the image acquisition unit has a built-in serializer, as... Figure 3 The image acquisition unit can directly output a GMSL format video stream to the first transcoder based on the continuously acquired panoramic images; then the first transcoder converts the received video stream into MIPI CSI-2 format and outputs it to the second transcoder; then the second transcoder converts the received video stream back into GMSL format and outputs it to the cockpit domain controller, so that the cockpit domain controller can receive the GMSL format video stream and decode it correctly during the SoC startup phase.

[0057] The image acquisition unit can include four surround-view cameras. All four video streams from the image acquisition unit are transmitted to the cockpit domain controller through the first transcoder and the second transcoder. The cockpit domain controller can decode the four video streams into video frames, and perform distortion correction and stitching on the four video frames acquired at the same time to generate a panoramic surround-view image.

[0058] Based on the above introduction, it can be seen that during the cold start phase, the interactive process of realizing the AVM function under the control of the first controller (i.e., the MCU) can be as follows: Figure 4 As shown, Figure 4 In this invention, "camera" represents the image acquisition unit, and "serdes" is an abbreviation for Serializer / Deserializer. To maintain compatibility with the original system architecture, the first transcoder (i.e., the deserializer) is directly connected to the cockpit domain controller via a newly added second transcoder (i.e., the serializer). This allows for the establishment of a video stream transmission channel from the image acquisition unit, deserializer, serializer, and cockpit domain controller under the control of the first controller during the second controller's startup phase. This eliminates the need to wait for the second controller to complete startup and enables rapid AVM (Area Virtual Machine) startup. Furthermore, because the first controller configures both the first and second transcoders beforehand, the cockpit domain controller can correctly decode the received video stream, resulting in a panoramic surround-view image displayed on the vehicle's central control screen without frame drops, misalignment, screen tearing, or complete signal loss.

[0059] In optional implementations, in addition to the surround-view camera required for the AVM function, the vehicle may also be equipped with other cameras (such as front-view cameras and rear-view cameras) and sensing components such as LiDAR. These components are typically configured and enabled uniformly after the SoC is booted up. To ensure the uniformity of the control entity, after the second controller is successfully booted up, the first controller of this invention will also transfer the control of the AVM function to the second controller, so that the second controller can uniformly manage the various sensing components of the vehicle.

[0060] Therefore, the method for generating the panoramic view image may further include the following steps S201 to S204.

[0061] S201. When the second controller completes the power-on startup, the second controller sends the first enable command and the acquisition control command to the transcoding channel unit and the image acquisition unit, respectively.

[0062] S202, The image acquisition unit acquires video streams representing the external environment of the vehicle based on acquisition control commands and transmits them to the transcoding channel unit.

[0063] S203, after the transcoding channel unit switches to the working mode based on the first enable command, it performs format conversion on the received video stream and transmits the converted video stream to the cockpit controller.

[0064] S204, the cockpit domain controller generates a panoramic surround view image based on the video stream.

[0065] In this embodiment, once the second controller completes power-on startup, it can take over from the first controller and take the lead in controlling the AVM function. That is, the second controller starts controlling the image acquisition unit to acquire video streams and transmits them to the cockpit domain controller through the transcoding channel unit.

[0066] Optionally, if the second controller needs video stream data captured by the surround-view camera, the image acquisition unit can also transmit the converted video stream to the second controller.

[0067] In one example, when the second controller starts successfully, it can notify the first controller to stop controlling the image acquisition unit to acquire video streams. Specifically, it can send a start success signal to the first controller. When the first controller receives the start success signal, it stops sending acquisition control commands to the surround view camera of the image acquisition unit. Thus, when the image acquisition unit does not receive acquisition control commands, it will stop acquiring video streams.

[0068] In another example, after the second controller starts successfully, once the image acquisition unit receives acquisition control commands from both the first and second controllers, it can assume that the acquisition control commands from the second controller have a higher priority than those from the first controller. In this case, the image acquisition unit will send a stop transmission signal to the first controller, and the first controller will stop sending acquisition control commands to the image acquisition unit upon receiving the stop transmission signal.

[0069] Optionally, to ensure time synchronization among the various sensing components mounted on the vehicle, the second controller needs to uniformly configure all sensing components upon startup before performing unified management, including the transcoding channel unit of this invention. That is, the method further includes step S200.

[0070] S200: After the second controller controls the transcoding channel unit to perform configuration reset, it takes over the transcoding channel unit for configuration and saves the corresponding configuration information when the takeover configuration is successful.

[0071] In this embodiment, after the second controller completes the power-on startup, it needs to execute step S200 first, and then execute steps S201~S204. This ensures that when the second controller takes the lead in controlling and implementing the AVM function, the time of the surround-view camera and other sensors is synchronized. The second controller must first reset the transcoding channel unit configuration before taking over the configuration, and then control the transcoding channel unit to switch to the working mode. Only then can the video stream transmitted by the subsequent image acquisition unit be correctly converted in format and transmitted to the cockpit domain controller.

[0072] Among them, combined Figure 3 The process of taking over the configuration can be as follows: the second controller sends a second enable command to the first transcoder and the second transcoder so that both the first transcoder and the second transcoder switch to low power mode based on the second enable command; then the second controller configures the first transcoder and the second transcoder respectively.

[0073] It's understandable, combined Figure 5 Once the second controller completes its power-on startup, it indicates that its operating system and application layer have started. The application layer of the second controller will then begin loading the image application and camera driver. Next, the image application will request the camera driver to open the surround-view camera. The camera driver will first check whether the initialization flag of the surround-view camera in the image acquisition unit is 1 from the shared memory of the first and second controllers. If the initialization flag of the surround-view camera is 1, the control takeover preparation phase will then begin. The camera driver sends a reset command to the first transcoder and the second transcoder, so that the first transcoder and the second transcoder will perform configuration initialization (i.e., configuration reset) based on the reset command.

[0074] Next, the camera driver sends a second enable command to the first transcoder and the second transcoder. Based on the second enable command, the first transcoder switches its output enable pin to low-power mode, i.e., LP (Low-Power) mode. The second transcoder also switches its input enable pin to low-power mode based on the second enable command.

[0075] Then the camera driver begins to configure the first transcoder and the second transcoder, and solidifies the configuration information. The specific configuration content is the same as the configuration content of the first controller mentioned above, and will not be repeated here.

[0076] Meanwhile, if the second controller needs the video stream from the surround-view camera, the camera driver also needs to initialize the VIO (Video IN / OUT) module inside the second controller to ensure that the VIO module is ready to receive the video stream from the surround-view camera. This VIO module includes the MIPI CSI-2 PHY chip (the physical layer chip that implements surround-view image processing) and the VIO interface.

[0077] Then, the camera driver can send a first enable command to the first and second transcoders via the I2C bus. Based on the first enable command, the first transcoder switches its output enable pin back to working mode, and the second transcoder switches its input enable pin back to working mode based on the first enable command. The camera driver then resets the initialization flag of the surround-view camera in shared memory to 1.

[0078] At this point, control takeover is complete, and the camera driver can begin sending acquisition control commands to the image acquisition unit. Thus, combined with... Figure 3 Under the control of the second controller, the image acquisition unit will also directly output a GMSL format video stream to the first transcoder; then the first transcoder converts the received video stream into MIPI CSI-2 format and outputs it to the second transcoder; then the second transcoder converts the received video stream back into GMSL format and outputs it to the cockpit domain controller to generate a panoramic surround view image.

[0079] Based on the above description, after the second controller starts up, the interaction process by which the second controller takes over to implement the AVM function can be as follows: Figure 5 As shown, Figure 5 The camera driver in the text refers to the camera driver, and the parallel video stream refers to the MIPI CSI-2 format video stream.

[0080] In this invention, upon power-on, the second controller first checks the initialization flag, then prepares for takeover, and finally takes over to implement the AVM function. During the takeover preparation, the first transcoder and the second converter are configured and reset, switching to a low-power mode for configuration and operation. This ensures that the subsequent data transmission between the two transcoders and other sensing components is synchronized. The second controller then takes over to implement the AVM function, enabling unified management of the acquisition control of all sensing components. Furthermore, after takeover, the second controller still uses the same video stream transmission channels from the image acquisition unit, deserializer, serializer, and cockpit domain controller, achieving a smooth transition of the AVM function from being dominated by the first controller to being dominated by the second controller.

[0081] It is understandable that, depending on the car configuration, the AVM function can be used when the car enters or exits the parking space (i.e., parking), and can also be used throughout the entire driving process (i.e., while the car is in motion, the in-vehicle central control screen displays a panoramic view image to help the driver judge the surrounding driving environment).

[0082] In the optional implementation, configuration failures may sometimes occur when configuring the first or second transcoder. This may be due to transient failures or bus conflicts. To improve system robustness, multiple retries can be allowed.

[0083] Therefore, taking the configuration end as the first controller or the second controller as an example, the method for generating the panoramic surround view image during the configuration process of the first transcoder or the second transcoder may also include the following steps a1 to a2: Step a1: When the configuration terminal receives a configuration failure signal from the first transcoder or the second transcoder, it resets the configuration of the object that reported the error and then reconfigures it.

[0084] Step a2: When the configuration terminal receives configuration failure signals for the same error object N times consecutively, it records the fault information.

[0085] In this embodiment, if an error occurs during the configuration of the first or second transcoder, a configuration failure signal, such as an INT (Interrupt) signal or an ERR (Error) signal, will be reported to the configuration terminal. The configuration terminal can then reset the configuration information of the object reporting the error by sending a reset signal, and then reconfigure. A maximum of N reconfiguration attempts (e.g., 2 or 3 times) are allowed. If N consecutive configuration failures occur, the fault information must be recorded and reported to the fault center.

[0086] One possible scenario is that when the car arrives at its destination and is parked, the user will use the AVM function implemented by the second controller. After parking, if the user puts the car in P gear, leaves the car and locks it, the cockpit domain controller will go into sleep mode, and the first controller will also directly control the various parts in the intelligent driving domain controller to go into sleep mode, and then the first controller will also go into sleep mode.

[0087] Another possible scenario is that when the car arrives at its destination and parks, the user utilizes the AVM function implemented by the second controller. After parking, the user can operate the in-vehicle central control screen or the control software on their mobile phone to activate the car's sentry mode, then shift the car into Park, leave the car, and lock it. During the parking period, the image acquisition unit continuously monitors the surrounding environment. During the sentry mode activation phase, this invention also transfers control of the AVM function from the second controller back to the first controller. This allows the first controller to control the second controller to directly enter sleep mode, achieving low-power monitoring.

[0088] Therefore, after the above steps S201 to S204, the method for generating the panoramic view image may further include the following steps S301 to S303.

[0089] When the S301 cockpit domain controller receives the monitoring start signal and the parking signal, it sends a low-power monitoring signal to the first controller.

[0090] In this embodiment, when the user activates the Sentinel mode, the cockpit domain controller receives a monitoring start signal. When the user shifts the car into Park, the cockpit domain controller receives a parking signal. If the cockpit domain controller receives both the monitoring start signal and the parking signal in sequence, it will send a low-power monitoring signal to the first controller.

[0091] S302. When the first controller receives a low-power monitoring signal, it controls the second controller to go into sleep mode.

[0092] It is understandable that the power consumption of the second controller is higher than that of the first controller because the second controller needs to run a large number of application add-ons. Therefore, in this embodiment, when the first controller receives a low-power monitoring signal, it can directly control the second controller to go into sleep mode, that is, control the PMIC to provide low-power power to the second controller. This can reduce the power consumption of the second controller, and the second controller directly releases control of the AVM function.

[0093] In this embodiment, after the second controller goes into sleep mode, the first controller can take over the AVM function. Optionally, after the second controller goes into sleep mode, the first controller also needs to prepare for takeover before it can officially take over (i.e., execute the above step S100). This process is similar to... Figure 5 Similar to the above, without going into detail here, the above steps S101~S104 are then executed. However, at this time, the cockpit domain controller will not display the generated panoramic surround view image on the vehicle's central control screen. Instead, it can perform target detection, such as detecting whether there are moving objects. If so, the panoramic surround view video of the most recent period of time will be pushed to the user.

[0094] In the optional implementation process, after the second controller starts up, during the handover of control of the surround view camera between the first controller and the second controller, the display of the generated panoramic surround view image will inevitably be interrupted, resulting in a brief black screen. Although the black screen time is relatively short (millisecond level), such a short black screen flicker will still affect the user experience when the user is using it.

[0095] Therefore, during the handover of control of the surround-view camera between the first controller and the second controller, this invention can display the last few frames of the video stream received by the cockpit domain controller, thus avoiding the visual perception of a disconnection or black screen for the user. Specifically, the method for generating the panoramic surround-view image may further include steps b1 to b3.

[0096] Step b1: After the second controller completes the power-on startup, the second controller sends a startup success signal to the first controller.

[0097] Step b2: When the first controller receives the start-up success signal, it sends a control switch signal to the cockpit domain controller.

[0098] Step b3: After receiving the control switch signal, the cockpit domain controller generates a panoramic surround view image based on the vehicle status and at least the last video frame of the video stream.

[0099] In one possible scenario, if the user starts the car, puts it in Park, and does not immediately press the accelerator to leave the parking space, then the current state of the car during the cold start phase is stationary. At this time, the cockpit domain controller can generate a panoramic surround view image based on the last video frame until the video stream is received again. Since the car is not moving at this time, directly displaying the last video frame will not make the user feel that the picture is choppy.

[0100] In another possible scenario, if the user gets into the car, starts it, puts it in Park, and immediately accelerates to leave the parking space, the car will be in a low-speed motion state during the cold start phase. If the last video frame is still displayed, the user will intuitively feel that the screen is choppy, which may lead to misjudgment or even operational risks.

[0101] In this scenario, the cockpit domain controller needs to continuously generate panoramic surround-view images using an optical flow algorithm based on the last N video frames of the received video stream (N can be 2, 3, or 5, etc.) until the video stream is received again. This allows the cockpit domain controller to determine the optical flow motion pattern based on the last N video frames, thereby simulating and generating multiple panoramic surround-view images for display. This results in a smooth transition of dynamic images on the in-vehicle central control screen, effectively bridging the visual gap during the handover of control. Furthermore, this method aligns with the vehicle's motion state, and the brief simulation does not affect the user's judgment of the surrounding driving environment.

[0102] Optionally, when the second controller begins controlling the image acquisition unit to perform image acquisition (i.e., when it begins sending acquisition control commands to the image acquisition unit), the second controller can use SOME / IP (Scalable Service-Oriented Middleware over IP, an Ethernet-based implementation) to... middleware The communication protocol sends a handover success signal to the cockpit domain controller, which then resumes receiving video streams to generate panoramic surround view images.

[0103] Optionally, if the second controller fails to initialize due to hardware failure or software error during the cold start phase, the second controller will send a start failure signal to the first controller. In this case, the first controller will continue to control the AVM function during the vehicle start-up and will not switch to SoC control, thereby maximizing the availability of the vehicle's AVM function.

[0104] It should be noted that the execution order of each step in the above method embodiments is not limited to the order shown in the attached figures and the step numbering, and the execution order of each step shall be subject to the actual application.

[0105] Based on the panoramic surround view image generation method described above, this embodiment of the invention also provides an intelligent driving system. The intelligent driving system includes an intelligent driving domain controller, a cockpit domain controller, and an image acquisition unit. The intelligent driving domain controller includes a first controller, a second controller, and a transcoding channel unit. The first controller is connected to the second controller, the image acquisition unit, and the transcoding channel unit. The second controller is connected to both the image acquisition unit and the transcoding channel unit. The transcoding channel unit is also connected to the cockpit domain controller. Wherein: When the first controller completes power-on startup but the second controller does not, the first controller is used to send a first enable command and an acquisition control command to the transcoding channel unit and the image acquisition unit, respectively. The image acquisition unit is used to acquire video streams representing the external environment of the vehicle based on acquisition control commands, and transmit them to the transcoding channel unit; The transcoding channel unit is used to convert the format of the received video stream after switching to the working mode based on the first enable command, and then transmit the converted video stream to the cockpit controller. The cockpit domain controller is used to generate panoramic surround view images based on the video stream.

[0106] The working principles of each part of the intelligent driving system in implementing the AVM function are described above and will not be repeated here.

[0107] This invention also provides a vehicle that includes the intelligent driving system described above.

[0108] In summary, this invention provides a method for generating a panoramic surround view image, an intelligent driving system, and a vehicle. The method involves the following steps: when the first controller has completed its power-on startup but the second controller has not, the first controller sends a first enable command and an acquisition control command to the transcoding channel unit and the image acquisition unit, respectively. The image acquisition unit, based on the acquisition control command, acquires a video stream representing the external environment of the vehicle and transmits it to the transcoding channel unit. After switching to its working mode based on the first enable command, the transcoding channel unit performs format conversion on the received video stream and transmits the converted video stream to the cockpit controller. The cockpit domain controller generates a panoramic surround view image based on the video stream. In this invention, during the power-on startup phase of the second controller, the first controller is responsible for controlling and implementing the AVM function. This directly establishes a data channel from the image acquisition unit, through the transcoding channel unit, to the cockpit domain controller, allowing the cockpit domain controller to quickly generate a panoramic surround view image based on the received video stream. During cold starts, there is no need to wait for the second controller to complete startup, and users can quickly use the AVM function.

[0109] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating a panoramic surround view image, characterized in that, The method includes: When the first controller completes power-on startup but the second controller does not complete power-on startup, the first controller sends a first enable command and an acquisition control command to the transcoding channel unit and the image acquisition unit, respectively. The image acquisition unit acquires a video stream representing the external environment of the vehicle based on the acquisition control command, and transmits it to the transcoding channel unit; After the transcoding channel unit switches to working mode based on the first enable command, it performs format conversion on the received video stream and transmits the converted video stream to the cockpit controller. The cockpit domain controller generates a panoramic surround view image based on the video stream.

2. The method for generating a panoramic view image according to claim 1, characterized in that, Before the image acquisition unit acquires a video stream characterizing the external environment based on the acquisition control command, the method further includes: When the second controller completes power-on startup, the second controller sends the first enable command and the acquisition control command to the transcoding channel unit and the image acquisition unit, respectively.

3. The method for generating a panoramic view image according to claim 2, characterized in that, The method further includes: When the second controller completes the power-on startup, the second controller sends a startup success signal to the first controller; When the first controller receives the start-up success signal, it sends a control switch signal to the cockpit domain controller. After receiving the control switch signal, the cockpit domain controller generates the panoramic surround view image based on the vehicle status and at least the last video frame of the video stream.

4. The method for generating a panoramic view image according to claim 3, characterized in that, After receiving the control switch signal, the cockpit domain controller generates the panoramic surround view image based on the vehicle status and at least the last video frame of the video stream, including: If the car is stationary, the cockpit domain controller generates a panoramic surround view image based on the last video frame until the video stream is received again. If the vehicle is in a low-speed motion state, the cockpit domain controller continuously generates a panoramic surround view image based on the last N video frames of the video stream using an optical flow algorithm until the video stream is received again.

5. The method for generating a panoramic surround view image according to claim 2, characterized in that, After the second controller sends the first enable command and the acquisition control command to the transcoding channel unit and the image acquisition unit respectively, the method further includes: When the cockpit domain controller receives the monitoring start signal and the parking signal, it sends a low-power monitoring signal to the first controller. When the first controller receives the low-power monitoring signal, it controls the second controller to go into sleep mode. The first controller sends a first enable command and an acquisition control command to the transcoding channel unit and the image acquisition unit, respectively.

6. The method for generating a panoramic surround view image according to claim 2, characterized in that, Before the first controller sends the first enable command and the acquisition control command to the transcoding channel unit and the image acquisition unit respectively, the method further includes: The first controller configures the transcoding channel unit to start the transcoding channel unit; The first controller saves the corresponding configuration information when the startup configuration is successful.

7. The method for generating a panoramic view image according to claim 6, characterized in that, Before the second controller sends the first enable command and the acquisition control command to the transcoding channel unit and the image acquisition unit respectively, the method further includes: The second controller controls the transcoding channel unit to perform a configuration reset; The second controller performs takeover configuration on the transcoding channel unit and saves the corresponding configuration information when the takeover configuration is successful.

8. The method for generating a panoramic view image according to claim 7, characterized in that, The transcoding channel unit includes a first transcoder and a second transcoder; the startup configuration includes: The first controller configures the first transcoder and the second transcoder respectively; The takeover configuration includes: The second controller sends a second enable command to the first transcoder and the second transcoder, so that both the first transcoder and the second transcoder switch to low power mode based on the second enable command; The second controller configures the first transcoder and the second transcoder respectively.

9. An intelligent driving system, characterized in that, The intelligent driving system includes an intelligent driving domain controller, a cockpit domain controller, and an image acquisition unit. The intelligent driving domain controller includes a first controller, a second controller, and a transcoding channel unit. The first controller is connected to the second controller, the image acquisition unit, and the transcoding channel unit. The second controller is connected to both the image acquisition unit and the transcoding channel unit. The transcoding channel unit is also connected to the cockpit domain controller. Wherein: When the first controller completes power-on startup but the second controller does not complete power-on startup, the first controller is used to send a first enable command and an acquisition control command to the transcoding channel unit and the image acquisition unit, respectively. The image acquisition unit is used to acquire a video stream representing the external environment of the vehicle based on the acquisition control command, and transmit it to the transcoding channel unit; The transcoding channel unit is used to switch to the working mode based on the first enable command, convert the format of the received video stream, and transmit the converted video stream to the cockpit controller. The cockpit domain controller is used to generate a panoramic surround view image based on the video stream.

10. A car, characterized in that, Including the intelligent driving system as described in claim 9.

Images