Image acquisition method, device, equipment and system

Abstract

The invention relates to an image acquisition method, device, equipment and system, and relates to the technical field of automobiles. The method comprises the steps that a fault monitoring module determines the fault state of a first image processing module and the fault state of a second image processing module; according to the fault state of the first image processing module and the fault state of the second image processing module, arbitration indication information is generated; the arbitration indication information is used for indicating a transmission path for acquiring the vehicle outside image data. Furthermore, the fault monitoring module sends arbitration indication information to the image string adding module, so that the image string adding module obtains the vehicle outer side image data according to a transmission path indicated by the arbitration indication information, and after the vehicle outer side image data is subjected to string adding, the vehicle outer side image data subjected to string adding is sent to a display screen of the vehicle. Therefore, the safety redundancy protection of the electronic outside rear-view mirror is realized.

Description

Technical Field

[0001] This application relates to the field of automotive technology, particularly to the field of safe driving technology for vehicles, and specifically to an image acquisition method, device, equipment, and system. Background Art

[0002] With the development of automotive intelligence, the electronic outside rearview mirror system is gradually replacing traditional optical rearview mirrors due to its broader field of view experience and more technological driving experience. The electronic outside rearview mirror system, through the combination of cameras, displays, and controllers, not only provides road condition information during driving but also reduces wind resistance and wind noise, and lowers the driving risks under strong and weak light conditions in rainy and snowy days.

[0003] However, due to the particularity of electronic circuit technology, it cannot guarantee 100% stable and reliable operation. Once there are significant delays or abnormal failures in the electronic outside rearview mirror control system, it will bring great risks to driving safety. Summary of the Invention

[0004] The purpose of this application is to provide an image acquisition method, device, equipment, and system for realizing the safety redundancy protection of electronic outside rearview mirrors.

[0005] To achieve the above purpose, the technical solution adopted in this application is as follows:

[0006] According to the first aspect of this application, there is provided an image acquisition method, which is applied to a fault monitoring module in an image acquisition system of a vehicle. The image acquisition system further includes a first image processing module, a second image processing module, and an image serialization module. The speed of the first image processing module for processing image data is greater than that of the second image processing module for processing image data. The method includes: the fault monitoring module determines the fault status of the first image processing module and the fault status of the second image processing module; and generates arbitration indication information based on the fault status of the first image processing module and the fault status of the second image processing module. The arbitration indication information is used to indicate the transmission path for acquiring vehicle outside image data. The transmission path includes a main transmission path or a backup transmission path. The main transmission path is the transmission path between the first image processing module and the image serialization module, and the backup transmission path is the transmission path between the second image processing module and the image serialization module. Further, the fault monitoring module sends the arbitration indication information to the image serialization module, so that the image serialization module acquires vehicle outside image data according to the transmission path indicated by the arbitration indication information, and after serializing the vehicle outside image data, sends the serialized vehicle outside image data to the display screen of the vehicle.

[0007] According to the above technical means, in the image acquisition method provided by the present application, through the fault monitoring module to monitor the faults of the first image processing module and the second image processing module, arbitration is performed according to the fault states of the first image processing module and the second image processing module to determine the transmission path for transmitting the vehicle exterior image data, so that the image concatenation module selects the corresponding transmission path, acquires the vehicle exterior image data for concatenation processing, and sends the concatenation-processed vehicle exterior image data to the display screen of the vehicle, and the display screen displays the vehicle exterior image data, realizing the safety redundancy protection of the electronic outside rearview mirror.

[0008] In a possible implementation manner, when the first image processing module is not in a fault state, the arbitration indication information is used to indicate that the transmission path for acquiring the vehicle exterior image data is the main transmission path; when the first image processing module is in a fault state and the second image processing module is not in a fault state, the arbitration indication information is used to indicate that the transmission path for acquiring the vehicle exterior image data is the backup transmission path.

[0009] According to the above technical means, the present application provides a way for the fault monitoring module to generate arbitration indication information for indicating the selection of the transmission path according to the fault states of the first image processing module and the second image processing module.

[0010] In a possible implementation manner, when the first image processing module is in a fault state, the first image processing module is restarted and reset; when the second image processing module is in a fault state, the second image processing module is restarted and reset.

[0011] In a possible implementation manner, the fault monitoring module determines the fault states of the first image processing module and the second image processing module, including: sending the first request message to the first image processing module multiple times and sending the second request message to the second image processing module multiple times; when the number of times of not receiving the first response message corresponding to the first request message is greater than the first preset number of times, determining that the first image processing module is in a fault state; when the number of times of not receiving the second response message corresponding to the second request message is greater than the second preset number of times, determining that the second image processing module is in a fault state.

[0012] According to the above technical means, the present application provides a specific implementation manner for the fault monitoring module to determine whether the first image processing module is in a fault state and to determine whether the second image processing module is in a fault state.

[0013] In a possible implementation, the fault monitoring module determines the fault status of the first image processing module and the fault status of the second image processing module, including: when receiving a first fault interrupt signal sent by the first image processing module, determining that the first image processing module is in a fault state; when receiving a second fault interrupt signal sent by the second image processing module, determining that the second image processing module is in a fault state.

[0014] According to the above technical means, the present application provides a specific implementation manner for the fault monitoring module to determine whether the first image processing module is in a fault state and to determine whether the second image processing module is in a fault state.

[0015] In a possible implementation, the fault monitoring module determines the fault status of the first image processing module, including: determining the first fault count of the first image processing module being in a fault state within a first preset time period; when the first fault count is greater than a third preset count, determining that the first image processing module has a hardware fault and sending a hardware fault reminder to the vehicle's display screen.

[0016] According to the above technical means, in the image acquisition method provided by the present application, by determining that the first image processing module has multiple faults, it is judged that the hardware of the first image processing module is damaged. To avoid the user from using the vehicle outer side image data transmitted by the second image processing module with a high delay for a long time, the user is reminded to repair the first image processing module in time to ensure the driving safety of the user.

[0017] In a possible implementation, the fault monitoring module determines the fault status of the second image processing module, including: determining the second fault count of the second image processing module being in a fault state within a second preset time period; when the second fault count is greater than a fourth preset count, determining that the second image processing module has a hardware fault and sending a hardware fault reminder to the vehicle's display screen.

[0018] In a possible implementation, the first image processing module is used to generate first outer image data, the second image processing module is used to generate second outer image data and advanced driver assistance system (ADAS) image data, and the fault monitoring module sends arbitration indication information to the image concatenation module, including: sending arbitration indication information to the image concatenation module when the ADAS function of the vehicle is in the off state; sending arbitration indication information to the image concatenation module when the ADAS function of the vehicle is in the on state; and when the arbitration indication information is used to indicate that the transmission path for obtaining the vehicle outer image data is the main transmission path, sending a first image fusion indication to the first image processing module, and when the arbitration indication information is used to indicate that the transmission path for obtaining the vehicle outer image data is the backup transmission path, sending a second image fusion indication to the second image processing module. The first image fusion indication is used to indicate that the first image processing module, after receiving the ADAS image data sent by the second image processing module, fuses the ADAS image data and the first outer image data to obtain first fused image data, and sends the first fused image data to the image concatenation module; the second image fusion indication is used to indicate that the second image processing module fuses the ADAS image data and the second outer image data to obtain second fused image data, and sends the second fused image data to the image concatenation module.

[0019] According to the above technical means, in the image acquisition method provided by the present application, when the ADAS function of the vehicle is in the on state, the vehicle outer image data is directly fused with the ADAS image data, which is more cost-saving in terms of hardware transmission compared to the ADAS image data being transmitted to the display screen through a controller area network (CAN) transceiver and a CAN bus and being mapped and displayed through an on-screen display (OSD) method.

[0020] In a possible implementation, the image acquisition system further includes an image deserialization module; the image deserialization module is used to deserialize the image data collected by the image acquisition devices on the left and right sides outside the vehicle after receiving the image data, and send the deserialized image data to the first image processing module and the second image processing module respectively; the first image processing module is used to process the deserialized image data after receiving the deserialized image data to obtain first outer image data, and the second image processing module is used to process the deserialized image data after receiving the deserialized image data to obtain second outer image data.

[0021] According to the above technical means, in the image acquisition method provided by the present application, based on the concurrent transmission characteristics of the image deserialization module, the first image processing module and the second image processing module can perform ISP imaging processing synchronously, so that the AI sub-module built in the second image processing module can perform parallel processing of AI image recognition on the ISP image data, realizing concurrent and synchronous processing of ADAS image recognition, without waiting for the first image processing module to complete imaging and then transmitting it to the second image processing module for AI image recognition, and the real-time performance of the ADAS function is higher.

[0022] In a possible implementation manner, the first image processing module is a field programmable gate array (FPGA) chip integrated with an image signal processing (ISP) algorithm, and the second image processing module is a system on chip (SOC) chip including an ISP sub-module and an artificial intelligence (AI) sub-module.

[0023] According to a second aspect provided by the present application, an image acquisition system is provided. The image acquisition system is deployed on a vehicle. The image acquisition system includes a fault monitoring module, a first image processing module, a second image processing module, an image concatenation module, and an image deconcatenation module. The speed at which the first image processing module processes image data is greater than the speed at which the second image processing module processes image data. The image deconcatenation module is configured to perform deconcatenation after receiving the image data sent by the image acquisition devices arranged on the left and right sides outside the vehicle, and send the deconcatenated image data to the first image processing module and the second image processing module respectively. The first image processing module is configured to process the deconcatenated image data after receiving it, obtain first outer side image data, and send the first outer side image data to the image concatenation module through the main transmission path. The second image processing module is configured to process the deconcatenated image data after receiving it, obtain second outer side image data, and send the second outer side image data to the image concatenation module through the backup transmission path. The fault monitoring module is configured to determine the fault status of the first image processing module and the fault status of the second image processing module. The fault monitoring module is further configured to generate arbitration indication information according to the fault status of the first image processing module and the fault status of the second image processing module. The arbitration indication information is used to indicate the transmission path for acquiring the vehicle outer side image data. The transmission path includes the main transmission path or the backup transmission path. The fault monitoring module is further configured to send the arbitration indication information to the image concatenation module. The image concatenation module is configured to acquire the vehicle outer side image data according to the transmission path indicated by the arbitration indication information. The image concatenation module is further configured to perform concatenation on the acquired vehicle outer side image data, and send the concatenated vehicle outer side image data to the display screen of the vehicle.

[0024] According to a third aspect provided by the present application, an image acquisition device is provided, which is a fault monitoring module deployed in an image acquisition system of a vehicle. The image acquisition system further includes a first image processing module, a second image processing module, and an image concatenation module. The speed of the first image processing module for processing image data is greater than that of the second image processing module for processing image data. The image acquisition device includes a determination unit, a processing unit, and a sending unit. The determination unit is configured to determine the fault status of the first image processing module and the fault status of the second image processing module. The processing unit is configured to generate arbitration indication information according to the fault status of the first image processing module and the fault status of the second image processing module. The arbitration indication information is used to indicate the transmission path for acquiring the image data outside the vehicle. The transmission path includes a main transmission path or a backup transmission path. The main transmission path is the transmission path between the first image processing module and the image concatenation module, and the backup transmission path is the transmission path between the second image processing module and the image concatenation module. The sending unit is configured to send the arbitration indication information to the image concatenation module, so that the image concatenation module acquires the image data outside the vehicle according to the transmission path indicated by the arbitration indication information, and after concatenating the image data outside the vehicle, sends the concatenated image data outside the vehicle to the display screen of the vehicle.

[0025] In a possible implementation manner, when the first image processing module is not in a fault state, the arbitration indication information is used to indicate that the transmission path for acquiring the image data outside the vehicle is the main transmission path. When the first image processing module is in a fault state and the second image processing module is not in a fault state, the arbitration indication information is used to indicate that the transmission path for acquiring the image data outside the vehicle is the backup transmission path.

[0026] In a possible implementation manner, the processing unit is further configured to restart and reset the first image processing module when the first image processing module is in a fault state. Restart and reset the second image processing module when the second image processing module is in a fault state.

[0027] In a possible implementation manner, the determination unit is specifically configured to send a first request message to the first image processing module multiple times and send a second request message to the second image processing module multiple times. When the number of times of not receiving the first response message corresponding to the first request message is greater than a first preset number of times, it is determined that the first image processing module is in a fault state. When the number of times of not receiving the second response message corresponding to the second request message is greater than a second preset number of times, it is determined that the second image processing module is in a fault state.

[0028] In a possible implementation, the determination unit is specifically configured to determine that the first image processing module is in a fault state when receiving a first fault interruption signal sent by the first image processing module. And determine that the second image processing module is in a fault state when receiving a second fault interruption signal sent by the second image processing module.

[0029] In a possible implementation, the determination unit is specifically configured to determine the first fault count of the first image processing module being in a fault state within a first preset duration; and determine that the first image processing module has a hardware fault and send a hardware fault reminder to the vehicle's display screen when the first fault count is greater than a third preset count.

[0030] In a possible implementation, the determination unit is specifically configured to determine the second fault count of the second image processing module being in a fault state within a second preset duration; and determine that the second image processing module has a hardware fault and send a hardware fault reminder to the vehicle's display screen when the second fault count is greater than a fourth preset count.

[0031] In a possible implementation, the first image processing module is an FPGA chip integrated with an ISP algorithm, and the second image processing module is an SOC chip including an ISP sub-module and an AI sub-module. The first image processing module is used to generate first outer image data based on the ISP algorithm, and the second image processing module is used to generate second outer image data based on the ISP sub-module. The second image processing module is further used to generate ADAS image data based on the AI sub-module. When the vehicle's ADAS function is in the off state, the sending unit is specifically configured to send arbitration indication information to the image concatenation module. When the vehicle's ADAS function is in the on state, the sending unit is specifically configured to send arbitration indication information to the image concatenation module; and when the arbitration indication information is used to indicate that the transmission path for obtaining the vehicle's outer image data is the main transmission path, send a first image fusion indication to the first image processing module, and when the arbitration indication information is used to indicate that the transmission path for obtaining the vehicle's outer image data is the backup transmission path, send a second image fusion indication to the second image processing module. The first image fusion indication is used to instruct the first image processing module to fuse the ADAS image data and the first outer image data after receiving the ADAS image data sent by the second image processing module to obtain first fusion image data and send the first fusion image data to the image concatenation module; the second image fusion indication is used to instruct the second image processing module to fuse the ADAS image data and the second outer image data to obtain second fusion image data and send the second fusion image data to the image concatenation module.

[0032] According to a fourth aspect provided by the present application, a microcontroller unit (MCU) is provided, which is deployed in a domain controller of a vehicle. The MCU includes a memory and a processor, and the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the MCU executes the image acquisition method provided by the first aspect and any possible implementation manner thereof.

[0033] According to a fifth aspect provided by the present application, a computer-readable storage medium is provided. Instructions are stored in the computer-readable storage medium. When the instructions run on the MCU, the MCU is caused to execute the image acquisition method provided by the first aspect and any possible implementation manner thereof.

[0034] According to a sixth aspect provided by the present application, a vehicle is provided, which includes the image acquisition system provided by the second aspect.

[0035] According to a seventh aspect provided by the present application, a computer program product is provided. The computer program product includes computer instructions. When the computer instructions run on the MCU, the MCU is caused to execute the image acquisition method provided by the first aspect and any possible implementation manner thereof.

[0036] Thus, the above technical features of the present application have the following beneficial effects:

[0037] (1) In the image acquisition method provided by the present application, through the failure monitoring of the first image processing module and the second image processing module by the failure monitoring module, arbitration is performed according to the failure status of the first image processing module and the failure status of the second image processing module to determine the transmission path for transmitting the vehicle outer side image data, so that the image serialization module can select the corresponding transmission path, acquire the vehicle outer side image data for serialization processing, and send the serialized vehicle outer side image data to the display screen of the vehicle, and the display screen displays the vehicle outer side image data, realizing the safety redundancy protection of the electronic outside rearview mirror.

[0038] It should be noted that the technical effects brought by any implementation manner in the second aspect to the seventh aspect can refer to the technical effects brought by the corresponding implementation manner in the first aspect, and will not be elaborated here.

[0039] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic structural diagram of an image acquisition system shown according to an exemplary embodiment;

[0041] Figure 2 is a flowchart of an image acquisition method shown according to an exemplary embodiment;

[0042] Figure 3 is a schematic diagram of the display of fused image data shown according to an exemplary embodiment;

[0043] Figure 4 is a schematic structural diagram of an electronic outside rearview mirror system shown according to an exemplary embodiment;

[0044] Figure 5 is a schematic diagram of signal transmission of an electronic outside rearview mirror system shown according to an exemplary embodiment;

[0045] Figure 6 is a flowchart of fault monitoring shown according to an exemplary embodiment;

[0046] Figure 7 is a flowchart of transmission path switching shown according to an exemplary embodiment;

[0047] Figure 8 is a flowchart of image fusion shown according to an exemplary embodiment;

[0048] Figure 9 is a block diagram of an image acquisition device shown according to an exemplary embodiment;

[0049] Figure 10 is a block diagram of an MCU shown according to an exemplary embodiment. Detailed implementation manners

[0050] The following will describe the implementation manners of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for explaining the present application, rather than for limiting the protection scope of the present application.

[0051] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner. Therefore, only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0052] In the description of the embodiments, unless otherwise specified, " / " means "or". For example, A / B may mean A or B. "And / or" in this article is merely a relationship describing associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, "at least one" and "multiple" mean two or more. The words such as "first" and "second" do not limit the quantity and execution order, and the words such as "first" and "second" do not necessarily limit to be different.

[0053] With the development of vehicle intelligence, the electronic outside rearview mirror system is gradually replacing the traditional optical rearview mirror due to its wider field of view experience and more technological driving experience. The electronic outside rearview mirror system, through the combination of cameras, displays, and controllers, not only provides road condition information during driving but also reduces wind resistance and wind noise and decreases the driving risks under strong and weak light conditions in rainy and snowy days. However, due to the particularity of electronic circuit technology, it cannot guarantee its stable and reliable operation 100%. Once there are large time delays or abnormal failures in the electronic outside rearview mirror control system, it will bring great risks to driving safety. In related technologies, most electronic outside rearview mirror systems adopt an independent controller solution, and the additional controller is expensive and causes many inconveniences to the overall vehicle space layout.

[0054] To solve the above problems, the present application proposes an image acquisition method, device, equipment, and system. The method is applied to a fault monitoring module in an image acquisition system of a vehicle. The image acquisition system further includes a first image processing module, a second image processing module, and an image concatenation module. The speed of the first image processing module for processing image data is greater than that of the second image processing module for processing image data. The method includes: the fault monitoring module determines the fault status of the first image processing module and the fault status of the second image processing module; and generates arbitration indication information according to the fault status of the first image processing module and the fault status of the second image processing module. The arbitration indication information is used to indicate the transmission path for acquiring vehicle outside image data. The transmission path includes a main transmission path or a backup transmission path. The main transmission path is the transmission path between the first image processing module and the image concatenation module, and the backup transmission path is the transmission path between the second image processing module and the image concatenation module. Further, the fault monitoring module sends the arbitration indication information to the image concatenation module, so that the image concatenation module acquires vehicle outside image data according to the transmission path indicated by the arbitration indication information, and after concatenating the vehicle outside image data, sends the concatenated vehicle outside image data to the display screen of the vehicle.

[0055] In this way, in the image acquisition method provided by this application, through the failure monitoring module's monitoring of the failures of the first image processing module and the second image processing module, arbitration is performed based on the failure states of the first image processing module and the second image processing module to determine the transmission path for transmitting the vehicle exterior image data, so that the image concatenation module can select the corresponding transmission path, acquire the vehicle exterior image data for concatenation processing, and send the concatenation-processed vehicle exterior image data to the vehicle's display screen, and the display screen displays the vehicle exterior image data, realizing the safety redundancy protection for the electronic outside rearview mirror.

[0056] Figure 1 An image acquisition system is shown. The image acquisition method provided by the embodiments of this application can be applicable to an image acquisition system as Figure 1 shown, for realizing the safety redundancy protection of the electronic outside rearview mirror. As Figure 1 shown, the image acquisition system 10 includes a failure monitoring module 11, a first image processing module 12, a second image processing module 13, an image concatenation module 14, and an image deconcatenation module 15.

[0057] Among them, the image acquisition system 10 is deployed in a vehicle. The failure monitoring module 11 is respectively connected to the first image processing module 12, the second image processing module 13, and the image concatenation module 14, and the image deconcatenation module 15 is respectively connected to the first image processing module 12 and the second image processing module 13.

[0058] It should be noted that both the first image processing module 12 and the second image processing module 13 are used to generate the vehicle exterior image data according to the image data collected by the image acquisition devices (electronic outside rearview mirrors) on the left and right sides outside the vehicle, and the speed at which the first image processing module 12 generates the vehicle exterior image data is greater than the speed at which the second image processing module generates the exterior image data.

[0059] The image deconcatenation module 15 is used to perform deconcatenation after receiving the image data sent by the image acquisition devices on the left and right sides outside the vehicle, and send the deconcatenated image data to the first image processing module 12 and the second image processing module 13 respectively;

[0060] The first image processing module 12 is used to process the deconcatenated image data after receiving it, obtain the first exterior image data, and send the first exterior image data to the image concatenation module 14 through the main transmission path;

[0061] The second image processing module 13 is used to process the deconcatenated image data after receiving it, obtain the second exterior image data, and send the second exterior image data to the image concatenation module 14 through the backup transmission path;

[0062] The fault monitoring module 11 is used to determine the fault status of the first image processing module 12 and the fault status of the second image processing module 13;

[0063] The fault monitoring module 11 is further used to generate arbitration indication information according to the fault status of the first image processing module 12 and the fault status of the second image processing module 13, and the arbitration indication information is used to indicate the transmission path for acquiring the vehicle outer side image data, and the transmission path includes a main transmission path or a standby transmission path;

[0064] The fault monitoring module 11 is further used to send the arbitration indication information to the image adding string module 14;

[0065] The image adding string module 14 is used to acquire the vehicle outer side image data according to the transmission path indicated by the arbitration indication information;

[0066] The image adding string module 14 is further used to add a string to the acquired vehicle outer side image data and send the vehicle outer side image data after adding the string to the display screen of the vehicle.

[0067] Figure 2 is a schematic flowchart of an image acquisition method shown according to some exemplary embodiments. In some embodiments, the above image acquisition method can be applied to the fault monitoring module 11 in the image acquisition system 10 as shown in Figure 1 The following takes the application of the image acquisition method to the fault monitoring module 11 as an example in the embodiments of the present application to illustrate the above image acquisition method.

[0068] As Figure 2 shown, the image acquisition method provided in the embodiments of the present application includes the following S201 - S203.

[0069] S201. The fault monitoring module determines the fault status of the first image processing module and the fault status of the second image processing module.

[0070] As a possible implementation manner, the fault monitoring module sends a first request message to the first image processing module multiple times and determines whether a first response message sent by the first image processing module in response to the first request message is received within a preset time period. The fault monitoring module determines the number of first response messages not received, and determines that the first image processing module is in a fault state when the number is greater than a first preset number.

[0071] The fault monitoring module also sends a second request message to the second image processing module multiple times and determines whether a second response message sent by the second image processing module in response to the second request message is received within a preset time period. The fault monitoring module determines the number of second response messages not received, and determines that the first image processing module is in a fault state when the number is greater than a second preset number.

[0072] Specifically, the fault monitoring module can periodically send a first request message to the first image processing module and a second request message to the second image processing module. Before sending the next request message, if the fault monitoring module does not receive the corresponding response message, it determines that the response message sent by the image processing module has not been received and counts the number of times the response message has not been received.

[0073] In some embodiments, taking the first image processing module as an example, when the fault monitoring module does not receive the first response message in a consecutive first preset number of cycles, it determines that the first image processing module is in a fault state.

[0074] In some embodiments, the first image processing module is configured to report a first fault interrupt signal in case of a severe hardware error, and the second image processing module is configured to report a second fault interrupt signal when the operating system kernel crashes.

[0075] When the fault monitoring module receives the first fault interrupt signal sent by the first image processing module, it determines that the first image processing module is in a fault state.

[0076] When the fault monitoring module receives the second fault interrupt signal sent by the second image processing module, it determines that the second image processing module is in a fault state.

[0077] In some embodiments, the fault monitoring module determines the first fault count of the first image processing module being in a fault state within a first preset duration. When the first fault count is greater than a third preset number, it determines that the first image processing module has a hardware fault and sends a hardware fault reminder to the vehicle's display screen, so that the display screen shows that the first image processing module has a hardware fault, reminding the user to repair the first image processing module.

[0078] In some embodiments, the fault monitoring module determines the second fault count of the second image processing module being in a fault state within a second preset duration. When the second fault count is greater than a fourth preset number, it determines that the second image processing module has a hardware fault and sends a hardware fault reminder to the vehicle's display screen, so that the display screen shows that the second image processing module has a hardware fault, reminding the user to repair the second image processing module.

[0079] It should be noted that the above first preset number, second preset number, third preset number, fourth preset number, as well as the first preset duration and the second preset duration, can be set in advance in the fault monitoring module by the operation and maintenance personnel of the image acquisition system. The embodiments of the present application do not make specific limitations on this.

[0080] In some embodiments, since the first image processing module and the second image processing module are each set with a waiting duration for restart and reset, before the first image processing module and the second image processing module are fully started, it may cause the fault monitoring module to misjudge that they are in a fault state. Therefore, in order to ensure the stability of the arbitration indication information generated subsequently and avoid frequent switching of the transmission path, in the image acquisition method provided by the embodiments of the present application, after the waiting duration set by the first image processing module, the fault monitoring module determines whether the first image processing module is in an abnormal state, and after the waiting duration set by the second image processing module, the fault monitoring module determines whether the second image processing module is in an abnormal state.

[0081] In some embodiments, to ensure the operation of the image acquisition system, when the fault monitoring module determines that the first image processing module is in a fault state, it restarts and resets the first image processing module, and after the waiting duration set by the first image processing module, continues to monitor whether the first image processing module is in an abnormal state. When the fault monitoring module determines that the second image processing module is in a fault state, it restarts and resets the second image processing module, and after the waiting duration set by the second image processing module, continues to monitor whether the second image processing module is in an abnormal state.

[0082] S202. The fault monitoring module generates arbitration indication information according to the fault state of the first image processing module and the fault state of the second image processing module.

[0083] Among them, the arbitration indication information is used to indicate the transmission path for acquiring the vehicle outer side image data. The transmission path includes a main transmission path or a standby transmission path. The main transmission path is the transmission path between the first image processing module and the image adding and stringing module, and the standby transmission path is the transmission path between the second image processing module and the image adding and stringing module.

[0084] As a possible implementation manner, since the speed at which the first image processing module generates the vehicle outer side image data of the vehicle is greater than the speed at which the second image processing module generates the vehicle outer side image data of the vehicle, when the fault monitoring module determines that the first image processing module is not in a fault state, the generated arbitration indication information is used to indicate that the transmission path for acquiring the vehicle outer side image data is the main transmission path; when it is determined that the first image processing module is in a fault state and the second image processing module is not in a fault state, the generated arbitration indication information is used to indicate that the transmission path for acquiring the vehicle outer side image data is the standby transmission path.

[0085] In some embodiments, the fault monitoring module runs two programs in parallel. One is that the fault monitoring module obtains the internal storage arbitration indication information and sends it to the image serialization module. The other is that the fault monitoring module generates arbitration indication information based on the fault status of the first image processing module and the fault status of the second image processing module, and updates the arbitration indication information stored internally after generating the arbitration indication information.

[0086] S203. The fault monitoring module sends the arbitration indication information to the image serialization module.

[0087] As a possible implementation, the fault monitoring module sends the arbitration indication information to the image serialization module based on the arbitration indication information generated in step S202 above.

[0088] Correspondingly, after receiving the arbitration indication information sent by the fault monitoring module, the image serialization module obtains the vehicle exterior image data according to the transmission path indicated by the arbitration indication information, and after serializing the vehicle exterior image data, sends the serialized vehicle exterior image data to the vehicle's display screen so that the vehicle's display screen displays the vehicle exterior image data.

[0089] In some embodiments, the first image processing module is an FPGA chip integrated with an ISP algorithm, and the second image processing module is an SOC chip including an ISP sub-module. The first image processing module is used to generate first exterior image data based on the ISP algorithm, and the second image processing module is used to generate second exterior image data based on the ISP sub-module.

[0090] The image acquisition system further includes an image deserialization module; the image deserialization module is used to deserialize the image data collected by the image acquisition devices on the left and right sides outside the vehicle after receiving it, and send the deserialized image data to the first image processing module and the second image processing module respectively.

[0091] Among them, the image deserialization module deserializes the image data sent by the image acquisition device in the form of low-voltage differential signaling (LVDS) to obtain the deserialized image data, and sends the deserialized image data to the first image processing module and the second image processing module respectively in the form of a mobile industry processor interface (MIPI) camera serial interface (CSI) signal.

[0092] Furthermore, the first image processing module is used to process the deserialized image data based on the ISP algorithm after receiving the deserialized image data to obtain the first exterior image data.

[0093] Among them, after receiving the deserialized image data sent by the image deserialization module in the form of MIPI CSI signals, the first image processing module performs fast imaging on the received MIPI CSI signals based on the ISP algorithm to obtain the first outer image data, and sends it to the image serialization module through the main transmission path in the form of MIPI display serial interface (DSI) signals.

[0094] The second image processing module is used to process the deserialized image data based on the ISP sub-module after receiving the deserialized image data to obtain the second outer image data.

[0095] Among them, after receiving the deserialized image data sent by the image deserialization module in the form of MIPI CSI signals, the second image processing module performs fast imaging on the received MIPI CSI signals based on the ISP sub-module to obtain the second outer image data, and sends it to the image serialization module through the backup transmission path in the form of MIPI DSI signals.

[0096] Correspondingly, after receiving the first outer image data sent by the first image processing module in the form of MIPI DSI signals through the main transmission path, or the second outer image data sent by the second image processing module in the form of MIPI DSI signals through the backup transmission path, the image serialization module performs serialization processing on the received image data to obtain the outer image data in the form of LVDS signals for display on the display screen, and sends the outer image data in the form of LVDS signals to the display screen of the vehicle, so that the display screen displays the outer image data.

[0097] In some embodiments, the second image processing module further includes an AI sub-module for generating ADAS image data based on the AI sub-module; when the ADAS function of the vehicle is in the on state, the image acquisition method provided by the embodiments of the present application further includes:

[0098] If the arbitration indication information is used to indicate that the transmission path for acquiring the outer image data of the vehicle is the main transmission path, the fault monitoring module sends a first image fusion indication to the first image processing module.

[0099] Among them, the first image fusion indication is used to indicate that the first image processing module, after receiving the ADAS image data sent by the second image processing module, fuses the ADAS image data and the first outer image data to obtain the first fused image data, and sends the first fused image data to the image serialization module.

[0100] It should be noted that when the arbitration instruction information is used to indicate that the transmission path for obtaining the vehicle outer side image data is the main transmission path, the fault monitoring module is further configured to send an ADAS data forwarding instruction to the second image processing module, so that after the second image processing module generates the ADAS image data based on the AI sub-module, it sends the ADAS image data to the first image processing module.

[0101] If the arbitration instruction information is used to indicate that the transmission path for obtaining the vehicle outer side image data is the backup transmission path, the fault monitoring module sends a second image fusion instruction to the second image processing module.

[0102] Wherein, the second image fusion instruction is used to instruct the second image processing module to fuse the ADAS image data and the second outer side image data to obtain the second fused image data, and send the second fused image data to the image string adding module.

[0103] Exemplarily, the fused image data can be a layer overlay between the ADAS image data and the vehicle outer side image data, realizing the display of the ADAS image data on the vehicle outer side image data, making the ADAS more vivid.

[0104] In addition, the fused image data can also be as Figure 3 shown, the display areas of the ADAS image data and the vehicle outer side image data are respectively set. Among them, the display area of the ADAS image data is located at the top of the display screen, and the rest is the display area of the vehicle outer side image data. By respectively setting and displaying the ADAS image data and the vehicle outer side image data in their respective display areas, it is possible to avoid the displayed images being too chaotic, resulting in the user being unable to obtain the road condition information and affecting the driving safety of the driver.

[0105] It can be understood that in the image acquisition method provided by the embodiments of the present application, through the fault monitoring of the first image processing module and the second image processing module by the fault monitoring module, arbitration is performed according to the fault status of the first image processing module and the fault status of the second image processing module to determine the transmission path for transmitting the vehicle outer side image data, so that the image string adding module selects the corresponding transmission path, acquires the vehicle outer side image data for string adding processing, and sends the vehicle outer side image data after the string adding processing to the display screen of the vehicle, and the display screen displays the vehicle outer side image data, realizing the safety redundancy protection of the electronic outside rearview mirror.

[0106] In one design, Figure 4The figure shows a schematic structural diagram of an electronic outside rearview mirror system, which includes an image acquisition device 31, a domain controller 32, and a display screen 33. Among them, the domain controller 32 includes an image deserialization module 321, a first image processing module 322, a second image processing module 323, a fault monitoring module 324, and an image serialization module 325.

[0107] The image acquisition device 31 can be the left and right cameras of the vehicle's electronic outside rearview mirror system.

[0108] The image acquisition device 31 can be used to acquire the outside image data of the vehicle (the road condition images on both sides of the vehicle body), and after serialization, it is converted into an LVDS signal and transmitted to the image deserialization module 321.

[0109] It should be noted that the image deserialization module 321 is a deserialization chip in the vehicle's domain controller. The deserialization chip can be DS90UB960, MAX96716, or other chips that support dual-channel MIPI CSI output ports. The deserialization chip needs to match the model of the serialization chip in the image acquisition device 31 to ensure the normal serial and deserial serial processing of the acquired image data.

[0110] The image deserialization module 321 can be used to deserialize the LVDS signal transmitted by the image acquisition device 31, and through clock channel replication technology, output two identical MIPI CSI signals, and concurrently transmit them to the first image processing module 322 and the second image processing module 323.

[0111] The first image processing module 322 can be an FPGA chip and its hard programming logic system in the vehicle's domain controller 32. The FPGA chip is an automotive-grade FPGA chip.

[0112] Among them, the FPGA chip quickly forms an image of the acquired image data by integrating ISP algorithms (including level compensation, automatic exposure, dynamic imaging, automatic white balance, color correction, denoising, etc.), and generates the outside image data of the vehicle.

[0113] The second image processing module 323 can be an SOC chip in the vehicle's domain controller 32 and the software system running on it. The SOC chip can be other automotive-grade SOC chips such as 8295, 8255, 8775, etc. with built-in ISP sub-modules.

[0114] The SOC chip incorporates an ISP sub-module and an AI sub-module. Among them, the ISP sub-module is used to image the acquired image data to generate the outer side image data of the vehicle, and the AI sub-module is used to perform AI image recognition based on the ISP imaging data to identify the position, size, and distance of preset target objects (such as motor vehicles, pedestrians, bicycles, obstacles, etc.). When it is recognized that the distance between the target object and the vehicle body is less than the set threshold, the second image processing module 323 generates ADAS image data such as warning icons and warning texts.

[0115] It should be noted that the image acquisition device 31 does not need to incorporate an ISP chip. The first image processing module 322 within the domain controller 32 of the vehicle integrates an ISP algorithm, and the second image processing module 323 incorporates an ISP sub-module, which can effectively replace the ISP chip function of the camera to achieve reuse and reduce the hardware costs of the left and right cameras.

[0116] The fault monitoring module 324 can be the MCU chip within the domain controller 32 of the vehicle and the software system running on it. The MCU chip is a vehicle-grade MCU chip.

[0117] The fault monitoring module 324 can determine the fault status of the first image processing module 322 and the second image processing module 323 through periodic monitoring and the interrupt signals actively reported by the first image processing module 322 and the second image processing module 323 when hardware faults occur.

[0118] The fault monitoring module 324 can also generate and save arbitration indication information according to the fault status of the first image processing module 322 and the second image processing module 323, and send the arbitration indication information to the image serialization module 325.

[0119] The image serialization module 325 can be the serialization chip within the domain controller 32 of the vehicle. The serialization chip can be DS90UB941, MAX96755, or other chips that support dual-channel MIPI DSI input.

[0120] It should be noted that the serialization chip needs to match the deserialization chip model built into the vehicle's display screen 33 to ensure the normal serial and deserial serial processing of the image display data.

[0121] The image serialization module 325 can achieve the switching of the transmission path through register configuration, and after receiving the vehicle outer side image data, serialize it and convert it into an LVDS signal, and send it to the display screen 33.

[0122] The display screen 33 can be the left and right display screens of the electronic outside rearview mirror system, including a deserialization chip, an LCD display driving module, etc.

[0123] The display screen 33 can be used for displaying vehicle exterior image data and ADAS image data.

[0124] In one design, Figure 5 A signal transmission schematic diagram of a vehicle electronic outside rearview mirror system is shown. The domain controller can be a cockpit domain controller, an intelligent driving domain controller, or a cabin-driving integrated domain controller, etc. In Figure 5 In the shown signal transmission schematic diagram, it includes a left camera 41, a right camera 42, a DES 43 (deserialization chip corresponding to the image deserialization module), an FPGA 44 (FPGA chip corresponding to the first image processing module), an SOC 45 (SOC chip corresponding to the second image processing module), an MCU 46, a SER 47 (serialization chip corresponding to the image serialization module), a left display screen 48, and a right display screen 49. Among them, an ISP sub-module 451 and an AI sub-module 452 are built into the SOC 45.

[0125] Among them, after the left camera 41 and the right camera 42 collect the road condition environment images on both sides of the vehicle body, they are serialized and transmitted to the deserialization chip DES 43 in the domain controller through LVDS signals.

[0126] After deserializing the LVDS signal, the deserialization chip DES 43 outputs two identical MIPI CSI signals through clock channel replication technology and concurrently transmits them to the FPGA 44 and the SOC 45 in the domain controller.

[0127] The SOC 45 can perform imaging and AI image recognition through the built-in ISP sub-module 451 and AI sub-module 452, and communicate with the FPGA 44 through I2C signals and MIPI DSI signals.

[0128] The SOC 45 sends the processed image data to the serialization chip SER 47 in the form of MIPI DSI signals.

[0129] The FPGA 44 performs fast imaging processing, and when the ADAS function of the vehicle is in the on state, it receives the I2C signals and MIPI DSI signals sent by the SOC 45 and performs image fusion.

[0130] The FPGA 44 sends the processed image data to the serialization chip SER 47 in the form of MIPI DSI signals.

[0131] The MCU 46 communicates with the FPGA 44 and the SOC 45 through SPI signals to monitor the abnormal fault states of the FPGA 44 and the SOC 45 and generate arbitration indication information. According to the arbitration indication information, the MCU 46 configures the registers of the serialization chip SER 47 through I2C signals to achieve the switching of the transmission path.

[0132] The serial addition chip SER47 switches the internal transmission path through the I2C signal sent by the MCU46, serializes the MIPI DSI signal sent by the FPGA44 or SOC45, and converts it into LVDS signals for transmission to the left display screen 48 and the right display screen 49 for display respectively.

[0133] In one design, in combination with the image acquisition method provided in the above embodiments of the present application, the specific implementation manner for the fault monitoring module to determine the fault status of the first image processing module and the fault status of the second image processing module is as Figure 6 shown in the fault monitoring flowchart shown, including S501 - S505.

[0134] S501. The fault monitoring module periodically sends request messages to the first image processing module and the second image processing module respectively.

[0135] S502. If the first image processing module and the second image processing module return response messages within a preset time period, it is regarded as a successful monitoring. When the number of consecutive successful monitoring times reaches their respective preset thresholds, the fault monitoring module determines that this image processing module is not in a fault state.

[0136] S503. If the first image processing module and the second image processing module do not return response messages in their respective preset time periods in a timely manner, it is regarded as a failed monitoring. When the number of consecutive failed monitoring times reaches their respective preset thresholds, the fault monitoring module determines that this image processing module is in a fault state.

[0137] S504. The fault monitoring module receives the fault interrupt signals reported by the first image processing module and the second image processing module. If the fault monitoring module receives a fault interrupt signal, it determines that this image processing module is in a fault state.

[0138] S505. The fault monitoring module sets a fault status counter for the first image processing module; if the count value of the fault status counter accumulates to the preset maximum threshold within a preset time period, it is determined that there is hardware damage to the first image processing module, and the user is notified to return the product to the factory for repair.

[0139] In one design, due to the complexity of the internal logic system of the first image processing module (i.e., the FPGA chip), timing anomalies may occur and lead to hardware failures when there are clock signal jitters or drifts, power supply under - voltage or over - voltage resulting in noise, and poor heat dissipation resulting in signal delays. To improve fault tolerance and reliability, in combination with the image acquisition method provided in the above embodiments of the present application, the specific implementation manner for realizing the secure redundancy protection of the transmission path is as Figure 7 shown in the transmission path switching flowchart shown, including S601 - S608.

[0140] S601. The fault monitoring module obtains the currently stored arbitration indication information, and based on the arbitration indication information, notifies the corresponding image processing module through the SPI signal to output the MIPI DSI signal to the image serialization module, configures the registers of the image serialization switch module through the I2C signal, and switches the transmission path to the transmission path indicated by the arbitration indication information.

[0141] S602. The fault monitoring module determines whether the first image processing module is in a fault state.

[0142] It should be noted that when the fault monitoring module determines that the first image processing module is in a fault state, it executes step S603; when it determines that the first image processing module is not in a fault state, it executes step S606.

[0143] S603. The fault monitoring module determines whether the second image processing module is in a fault state.

[0144] It should be noted that when the fault monitoring module determines that the second image processing module is in a fault state, it executes step S604; when it determines that the second image processing module is not in a fault state, it executes step S605.

[0145] S604. The fault monitoring module keeps the currently stored arbitration indication information unchanged, does not change the transmission path of the image serialization module, and restarts and resets the first image processing module and the second image processing module.

[0146] S605. The fault monitoring module generates and stores arbitration indication information for indicating the standby transmission path; sends a first SPI signal to the second image processing module to make the second image processing module output the MIPI DSI signal to the image serialization module; sends a first I2C signal to the image serialization module to make the image serialization module switch the transmission path to the standby transmission path; restarts and resets the first image processing module.

[0147] S606. The fault monitoring module determines whether the currently stored arbitration indication information is used to indicate the main transmission path.

[0148] It should be noted that when the fault monitoring module determines that the currently stored arbitration indication information is used to indicate the main transmission path, it executes step S607; when it determines that the currently stored arbitration indication information is used to indicate the standby transmission path, it executes step S608.

[0149] S607. The fault monitoring module keeps the currently stored arbitration indication information unchanged, and does not change the transmission path of the image serialization module.

[0150] S608. The fault monitoring module generates and stores arbitration indication information for indicating the main transmission path; sends a second SPI signal to the first image processing module to enable the first image processing module to output a MIPI DSI signal to the image concatenation module; sends a second I2C signal to the image concatenation module to enable the image concatenation module to switch the transmission path to the main transmission path.

[0151] In one design, in combination with the image acquisition method provided in the above embodiments of the present application, a specific implementation manner for realizing the fusion of ADAS image data and vehicle exterior image data is as Figure 8 shown in the image fusion flowchart shown, including S701 - S707.

[0152] S701. The second image processing module acquires whether the ADAS function of the vehicle is in an enabled state.

[0153] It should be noted that when the ADAS function is in an enabled state, the second image processing module executes step S702, and when the ADAS function is not in an enabled state, the second image processing module executes step S705.

[0154] S702. The second image processing module determines whether the arbitration indication information indicates the main transmission path.

[0155] It should be noted that when the arbitration indication information indicates the main transmission path, the second image processing module executes step S703, and when the arbitration indication information does not indicate the main transmission path, the second image processing module executes step S704.

[0156] S703. The second image processing module notifies the first image processing module to enable ADAS image data fusion through a third I2C signal; performs AI image recognition based on ISP image data to generate ADAS image data; transmits the ADAS image data to the first image processing module through a MIPI DSI signal.

[0157] S704. The second image processing module performs AI image recognition based on ISP image data to generate ADAS image data and fuses it with the ISP image data.

[0158] S705. The second image processing module determines whether the arbitration indication information indicates the main transmission path.

[0159] It should be noted that when the arbitration indication information indicates the main transmission path, the second image processing module executes step S706, and when the arbitration indication information does not indicate the main transmission path, the second image processing module executes step S707.

[0160] S706. The second image processing module notifies the first image processing module to turn off the ADAS image data fusion through the third I2C signal; and stops the ISP sub-module and the AI sub-module from processing the acquired image data.

[0161] S707. The second image processing module enables the ISP sub-module to perform imaging processing on the image acquisition data; and stops the AI sub-module from performing AI image recognition on the ISP imaging data.

[0162] The above mainly introduces the solution provided by the embodiments of the present application from the perspective of methods. To implement the above functions, the fault monitoring module includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0163] The embodiments of the present application can, according to the above method, exemplarily divide the function modules of the fault monitoring module. For example, the fault monitoring module can include each function module corresponding to each function division, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software function module. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0164] Figure 9 It is a schematic structural diagram of an image acquisition device provided by an embodiment of the present application. This image acquisition device is used to execute the above image acquisition method. As Figure 9 shown, the image acquisition device 80 includes a determination unit 801, a processing unit 802, and a sending unit 803.

[0165] The determination unit 801 is used to determine the fault status of the first image processing module and the fault status of the second image processing module.

[0166] A processing unit 802 is configured to generate arbitration indication information according to the fault status of the first image processing module and the fault status of the second image processing module; the arbitration indication information is used to indicate a transmission path for acquiring vehicle exterior image data, and the transmission path includes a main transmission path or a backup transmission path. The main transmission path is the transmission path between the first image processing module and the image concatenation module, and the backup transmission path is the transmission path between the second image processing module and the image concatenation module.

[0167] A sending unit 803 is configured to send the arbitration indication information to the image concatenation module, so that the image concatenation module acquires vehicle exterior image data according to the transmission path indicated by the arbitration indication information, and after concatenating the vehicle exterior image data, sends the concatenated vehicle exterior image data to the vehicle display screen.

[0168] Optionally, when the first image processing module is not in a fault state, the arbitration indication information is used to indicate that the transmission path for acquiring vehicle exterior image data is the main transmission path; when the first image processing module is in a fault state and the second image processing module is not in a fault state, the arbitration indication information is used to indicate that the transmission path for acquiring vehicle exterior image data is the backup transmission path.

[0169] Optionally, the processing unit 802 is further configured to restart and reset the first image processing module when the first image processing module is in a fault state. Restart and reset the second image processing module when the second image processing module is in a fault state.

[0170] Optionally, the determining unit 801 is specifically configured to send a first request message to the first image processing module multiple times and send a second request message to the second image processing module multiple times; when the number of times of not receiving the first response message corresponding to the first request message is greater than a first preset number of times, determine that the first image processing module is in a fault state; when the number of times of not receiving the second response message corresponding to the second request message is greater than a second preset number of times, determine that the second image processing module is in a fault state.

[0171] Optionally, the determining unit 801 is specifically configured to determine that the first image processing module is in a fault state when receiving a first fault interrupt signal sent by the first image processing module. Determine that the second image processing module is in a fault state when receiving a second fault interrupt signal sent by the second image processing module.

[0172] Optionally, the determining unit 801 is further configured to determine a first fault count of the first image processing module being in a fault state within a first preset duration; when the first fault count is greater than a third preset number of times, determine that the first image processing module has a hardware fault and send a hardware fault reminder to the vehicle display screen.

[0173] Optionally, the determination unit 801 is further configured to determine the second number of faults when the second image processing module is in a faulty state within a second preset duration; when the second number of faults is greater than a fourth preset number, it is determined that the second image processing module has a hardware fault, and a hardware fault reminder is sent to the vehicle's display screen.

[0174] Optionally, the first image processing module is an FPGA chip integrated with an ISP algorithm, and the second image processing module is an SOC chip including an ISP sub-module and an AI sub-module. The first image processing module is configured to generate first outer image data based on the ISP algorithm, the second image processing module is configured to generate second outer image data based on the ISP sub-module, and the second image processing module is further configured to generate ADAS image data based on the AI sub-module. When the ADAS function of the vehicle is in the off state, the sending unit 803 is specifically configured to send arbitration indication information to the image concatenation module.

[0175] When the ADAS function of the vehicle is in the on state, the sending unit 803 is specifically configured to send arbitration indication information to the image concatenation module; and when the arbitration indication information is used to indicate that the transmission path for obtaining the vehicle outer image data is the main transmission path, a first image fusion indication is sent to the first image processing module, and when the arbitration indication information is used to indicate that the transmission path for obtaining the vehicle outer image data is the backup transmission path, a second image fusion indication is sent to the second image processing module. The first image fusion indication is used to indicate that the first image processing module fuses the ADAS image data and the first outer image data after receiving the ADAS image data sent by the second image processing module to obtain first fusion image data, and sends the first fusion image data to the image concatenation module; the second image fusion indication is used to indicate that the second image processing module fuses the ADAS image data and the second outer image data to obtain second fusion image data, and sends the second fusion image data to the image concatenation module.

[0176] Figure 10 is a block diagram of an MCU shown according to an exemplary embodiment. As Figure 10 shown, the MCU 90 includes but is not limited to: a processor 901 and a memory 902.

[0177] Among them, the above-mentioned memory 902 is used to store the executable instructions of the above-mentioned processor 901. It can be understood that the above-mentioned processor 901 is configured to execute instructions to implement the image acquisition method in the above-mentioned embodiment.

[0178] It should be noted that those skilled in the art can understand that Figure 10 the MCU structure shown inFigure 10 more or fewer components as shown, or combine certain components, or different component arrangements.

[0179] The processor 901 is the control center of the MCU, connecting various parts of the entire MCU using various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 902, and by calling data stored in the memory 902, it performs various functions of the MCU and processes data, thereby monitoring the MCU as a whole. The processor 901 may include one or more processing units. Optionally, the processor 901 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 901 either.

[0180] The memory 902 can be used to store software programs and various data. The memory 902 mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one functional module (such as a determination unit, a processing unit, etc.), and so on. In addition, the memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0181] In an exemplary embodiment, there is also provided a computer-readable storage medium including instructions, such as the memory 902 including instructions. The above instructions can be executed by the processor 901 of the MCU90 to implement the image acquisition method in the above embodiment.

[0182] In actual implementation, Figure 9 the functions of the determination unit 801, the processing unit 802, and the sending unit 803 in Figure 10 can all be implemented by the processor 901 in

[0183] calling the computer program stored in the memory 902. The specific execution process can refer to the description of the image acquisition method part in the above embodiment, and will not be elaborated here.

[0184] In an exemplary embodiment, the embodiments of the present application also provide a vehicle including the above-mentioned MCU.

[0185] In an exemplary embodiment, the embodiment of the present application further provides a computer program product including one or more instructions, and the one or more instructions can be executed by the processor 901 of the MCU to complete the image acquisition method in the above embodiment.

[0186] It should be noted that when the instructions in the above computer-readable storage medium or the one or more instructions in the computer program product are executed by the processor of the MCU, each process of the above image acquisition method embodiment is implemented, and the same technical effect as the above image acquisition method can be achieved. To avoid repetition, it will not be elaborated here.

[0187] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0188] In several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0189] The unit described as a separated component may or may not be physically separated, and the component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0191] When an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0192] The above are only the specific implementation manners of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image acquisition method, characterized in that, A fault monitoring module in an image acquisition system applied to a vehicle. The image acquisition system further includes a first image processing module, a second image processing module, and an image concatenation module. The speed of the first image processing module for processing image data is greater than that of the second image processing module for processing image data. The method includes: Determine the fault status of the first image processing module and the fault status of the second image processing module; Generate arbitration indication information according to the fault status of the first image processing module and the fault status of the second image processing module. The arbitration indication information is used to indicate the transmission path for acquiring image data outside the vehicle. The transmission path includes a main transmission path or a backup transmission path. The main transmission path is the transmission path between the first image processing module and the image concatenation module, and the backup transmission path is the transmission path between the second image processing module and the image concatenation module; Send the arbitration indication information to the image concatenation module, so that the image concatenation module acquires the image data outside the vehicle according to the transmission path indicated by the arbitration indication information, and after concatenating the image data outside the vehicle, send the concatenated image data outside the vehicle to the display screen of the vehicle.

2. The image acquisition method according to claim 1, wherein When the first image processing module is not in a fault state, the arbitration indication information is used to indicate that the transmission path for acquiring the image data outside the vehicle is the main transmission path; When the first image processing module is in a fault state and the second image processing module is not in a fault state, the arbitration indication information is used to indicate that the transmission path for acquiring the image data outside the vehicle is the backup transmission path.

3. The image acquisition method according to claim 1 or 2, characterized in that, The method further includes: Restart and reset the first image processing module when the first image processing module is in a fault state; Restart and reset the second image processing module when the second image processing module is in a fault state.

4. The image acquisition method according to claim 1 or 2, wherein The determining the fault status of the first image processing module and the fault status of the second image processing module includes: Send a first request message to the first image processing module multiple times and send a second request message to the second image processing module multiple times; When the number of times of not receiving a first response message corresponding to the first request message is greater than a first preset number of times, determine that the first image processing module is in a fault state; When the number of times of not receiving a second response message corresponding to the second request message is greater than a second preset number of times, determine that the second image processing module is in a fault state.

5. The image acquisition method according to claim 1 or 2, characterized in that The determining the fault status of the first image processing module and the fault status of the second image processing module includes: When receiving a first fault interrupt signal sent by the first image processing module, determine that the first image processing module is in a fault state; When receiving a second fault interrupt signal sent by the second image processing module, determine that the second image processing module is in a fault state.

6. The image acquisition method according to claim 1 or 2, characterized in that, Determining the fault status of the first image processing module includes: Determine the first failure count of the first image processing module being in a failure state within a first preset duration; When the first failure count is greater than a third preset count, determine that the first image processing module has a hardware failure, and send a hardware failure reminder to the vehicle's display screen.

7. The image acquisition method according to claim 1 or 2, characterized in that, Determine the failure state of the second image processing module, including: Determine the second failure count of the second image processing module being in a failure state within a second preset duration; When the second failure count is greater than a fourth preset count, determine that the second image processing module has a hardware failure, and send a hardware failure reminder to the vehicle's display screen.

8. The image acquisition method according to claim 1 or 2, characterized in that, The first image processing module is used to generate first outer side image data, and the second image processing module is used to generate second outer side image data and Advanced Driving Assistance System (ADAS) image data. Sending the arbitration indication information to the image concatenation module includes: When the vehicle's ADAS function is in a closed state, send the arbitration indication information to the image concatenation module; When the vehicle's ADAS function is in an open state, send the arbitration indication information to the image concatenation module; and when the arbitration indication information is used to indicate that the transmission path for acquiring the vehicle's outer side image data is the main transmission path, send a first image fusion indication to the first image processing module, and when the arbitration indication information is used to indicate that the transmission path for acquiring the vehicle's outer side image data is the backup transmission path, send a second image fusion indication to the second image processing module; The first image fusion indication is used to indicate that the first image processing module, after receiving the ADAS image data sent by the second image processing module, fuses the ADAS image data and the first outer side image data to obtain first fused image data, and sends the first fused image data to the image concatenation module; the second image fusion indication is used to indicate that the second image processing module fuses the ADAS image data and the second outer side image data to obtain second fused image data, and sends the second fused image data to the image concatenation module.

9. The image acquisition method according to claim 8, wherein The image acquisition system further includes an image deserialization module; the image deserialization module is used to deserialize the image data collected by the image acquisition devices on the left and right sides outside the vehicle after receiving the image data, and send the deserialized image data to the first image processing module and the second image processing module respectively; the first image processing module is used to process the deserialized image data after receiving the deserialized image data to obtain the first outer side image data, and the second image processing module is used to process the deserialized image data after receiving the deserialized image data to obtain the second outer side image data.

10. The image acquisition method according to claim 1 or 2, characterized in that, The first image processing module is a Field Programmable Gate Array (FPGA) chip integrated with an Image Signal Processing (ISP) algorithm, and the second image processing module is a System on Chip (SOC) chip including an ISP sub-module and an Artificial Intelligence (AI) sub-module.

11. An image acquisition system, characterized in that, The image acquisition system is deployed on a vehicle. The image acquisition system includes a fault monitoring module, a first image processing module, a second image processing module, an image multiplexing module, and an image demultiplexing module. The speed of the first image processing module for processing image data is greater than that of the second image processing module for processing image data. The image demultiplexing module is configured to demultiplex the image data sent by the image acquisition devices arranged on the left and right sides outside the vehicle after receiving the image data, and send the demultiplexed image data to the first image processing module and the second image processing module respectively. The first image processing module is configured to process the demultiplexed image data after receiving the demultiplexed image data, obtain first outer side image data, and send the first outer side image data to the image multiplexing module through the main transmission path. The second image processing module is configured to process the demultiplexed image data after receiving the demultiplexed image data, obtain second outer side image data, and send the second outer side image data to the image multiplexing module through the backup transmission path. The fault monitoring module is configured to determine the fault status of the first image processing module and the fault status of the second image processing module. The fault monitoring module is further configured to generate arbitration indication information according to the fault status of the first image processing module and the fault status of the second image processing module. The arbitration indication information is used to indicate the transmission path for acquiring the vehicle outer side image data. The transmission path includes the main transmission path or the backup transmission path. The fault monitoring module is further configured to send the arbitration indication information to the image multiplexing module. The image multiplexing module is configured to acquire the vehicle outer side image data according to the transmission path indicated by the arbitration indication information. The image multiplexing module is further configured to multiplex the acquired vehicle outer side image data and send the multiplexed vehicle outer side image data to the display screen of the vehicle.

12. An image acquisition device, characterized in that, The fault monitoring module in the image acquisition system deployed on a vehicle. The image acquisition system further includes a first image processing module, a second image processing module, and an image multiplexing module. The speed of the first image processing module for processing image data is greater than that of the second image processing module for processing image data. The image acquisition device includes a determination unit, a processing unit, and a sending unit. The determination unit is configured to determine the fault status of the first image processing module and the fault status of the second image processing module. The processing unit is configured to generate arbitration indication information according to the fault status of the first image processing module and the fault status of the second image processing module. The arbitration indication information is used to indicate the transmission path for acquiring the vehicle outer side image data. The transmission path includes the main transmission path or the backup transmission path. The main transmission path is the transmission path between the first image processing module and the image multiplexing module, and the backup transmission path is the transmission path between the second image processing module and the image multiplexing module. The sending unit is configured to send the arbitration indication information to the image concatenation module, so that the image concatenation module acquires the vehicle exterior image data according to the transmission path indicated by the arbitration indication information, and after concatenating the vehicle exterior image data, sends the concatenated vehicle exterior image data to the display screen of the vehicle.

13. A computer-readable storage medium storing instructions therein, characterized in that, When the instruction runs on a microcontroller unit (MCU), the MCU chip executes the image acquisition method according to any one of claims 1-10.

14. A vehicle, characterized in that, It includes the image acquisition system according to claim 11.

Images