Vehicle cabin control system, image processing method and apparatus, vehicle, electronic device, storage medium, and computer program product

Abstract

The present disclosure provides a vehicle cabin control system, an image processing method and apparatus, a vehicle, an electronic device, a storage medium, and a computer program product. The cabin control system comprises a control device and an output device in communication connection with the control device. The control device comprises at least a first processor and a second processor in communication with the first processor. Each of the first processor and the second processor comprises a microkernel operating system and a macrokernel operating system. The output device is used for outputting a target recognition result. The target recognition result comprises a first recognition result and / or a second recognition result. The first recognition result is obtained by means of processing an acquired image using a perception algorithm in the microkernel operating system of the second processor. The second recognition result is obtained by means of processing the first recognition result using a target operating system. The target operating system is the macrokernel operating system of the second processor or the macrokernel operating system of the first processor.

Description

Vehicle cabin control system, image processing method and device, vehicle, electronic device, storage medium and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The embodiments of the present disclosure are based on and claim the priority of Chinese patent application with application number 202311102045.5, application date August 29, 2023, and application name “Cockpit control system, image processing method and device, vehicle, and electronic device”. The entire content of the Chinese patent application is hereby introduced into the present disclosure as a reference. Technical Field

[0003] The present disclosure relates to, but is not limited to, the field of vehicle technology, and in particular to a vehicle cabin control system, an image processing method and apparatus, a vehicle, an electronic device, a storage medium, and a computer program product. Background Art

[0004] In related technologies, in the smart cockpit system, in order to support functions such as DMS (Driver Monitoring System) and OMS (Occupancy Monitoring System), the trained model needs to be integrated into the vehicle computer. However, in the actual reasoning process, a lot of computing power is required and a large amount of memory is consumed, making it impossible for the vehicle computer to efficiently execute other services.

[0005] Summary of the Invention

[0006] Embodiments of the present disclosure provide a vehicle cabin control system, an image processing method and apparatus, a vehicle, an electronic device, a storage medium, and a computer program product.

[0007] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0008] An embodiment of the present disclosure provides a vehicle cabin control system, comprising a control device and an output device communicatively connected to the control device, wherein the control device includes at least a first processor and a second processor communicating with the first processor, wherein the first processor and the second processor both include a microkernel operating system and a macrokernel operating system; wherein:

[0009] The output device is used to output the target recognition result; wherein the target recognition result includes a first recognition result and / or a second recognition result, the first recognition result is obtained by processing the acquired image using the perception algorithm in the microkernel operating system of the second processor, and the second recognition result is obtained by processing the first recognition result using the target operating system, and the target operating system is the macrokernel operating system of the second processor or the macrokernel operating system of the first processor.

[0010] An embodiment of the present disclosure provides an image processing method, which is applied to a vehicle cockpit control system. The cockpit control system includes a control device, the control device including a first processor and a second processor communicating with the first processor, the first processor and the second processor both including a microkernel operating system and a macrokernel operating system. The method includes:

[0011] Using a perception algorithm in the microkernel operating system of the second processor, the acquired image is recognized to obtain a first recognition result;

[0012] Output the first recognition result and / or the second recognition result; wherein, the second recognition result is obtained by processing the first recognition result using a target operating system, and the target operating system is the macro kernel operating system of the second processor or the macro kernel operating system of the first processor.

[0013] An embodiment of the present disclosure provides an image processing apparatus for use in a vehicle cockpit control system. The cockpit control system includes a control device, the control device including a first processor and a second processor communicating with the first processor. The first processor and the second processor both include a microkernel operating system and a macrokernel operating system. The apparatus includes:

[0014] a recognition part configured to recognize the acquired image using a perception algorithm in the microkernel operating system of the second processor to obtain a first recognition result;

[0015] The output part is configured to output the first recognition result and / or the second recognition result; wherein, the second recognition result is obtained by processing the first recognition result using the target operating system, and the target operating system is the macro kernel operating system of the second processor or the macro kernel operating system of the first processor.

[0016] An embodiment of the present disclosure provides a vehicle, comprising any of the above-mentioned cockpit control systems.

[0017] An embodiment of the present disclosure provides an electronic device, including a processor and a memory, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above method when executing the computer program.

[0018] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above method is implemented.

[0019] An embodiment of the present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, the above method is implemented.

[0020] In an embodiment of the present disclosure, a cockpit control system includes a control device and an output device communicatively connected to the control device. The control device includes at least a first processor and a second processor communicating with the first processor. Both the first and second processors include a microkernel operating system and a macrokernel operating system. The output device is configured to output a target recognition result. The target recognition result includes a first recognition result and / or a second recognition result. The first recognition result is obtained by processing a captured image using a perception algorithm in the microkernel operating system of the second processor, and the second recognition result is obtained by processing the first recognition result using a target operating system, which may be the macrokernel operating system of the second processor or the macrokernel operating system of the first processor. Thus, by deploying a dual-core control device in the cockpit control system, deploying dual systems on each processor, and deploying the perception algorithm in the microkernel operating system of a processor, algorithm reasoning and business operations are separated from each other. This allows other business operations to be executed normally while algorithm reasoning is being executed, thereby improving business execution efficiency.

[0021] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0023] FIG1A is a schematic diagram of the structure of a cockpit control system provided by an embodiment of the present disclosure;

[0024] FIG1B is a schematic diagram of the structure of a vehicle cabin control system provided by an embodiment of the present disclosure;

[0025] FIG1C is a schematic diagram of the structure of a control device provided by an embodiment of the present disclosure;

[0026] FIG1D is a schematic diagram of the structure of a cockpit control system provided by an embodiment of the present disclosure;

[0027] FIG2 is a schematic diagram of an implementation flow of an image processing method provided by an embodiment of the present disclosure;

[0028] FIG3 is a schematic diagram of the composition structure of an image processing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0030] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] In the following description, the terms "first\second\third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art of the present disclosure. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0033] Below, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the drawings in the embodiments of the present disclosure.

[0034] FIG1A is a schematic diagram of the structure of a cockpit control system provided by an embodiment of the present disclosure. As shown in FIG1A , the cockpit control system 10 includes a control device 11 and an output device 12 communicatively connected to the control device, wherein:

[0035] The control device 11 includes at least a first processor 111 and a second processor 112 communicating with the first processor 111, wherein the first processor 111 and the second processor 112 both include a microkernel operating system and a macrokernel operating system;

[0036] The output device 12 is used to output the target recognition result, wherein the target recognition result includes a first recognition result and / or a second recognition result, the first recognition result is obtained by processing the acquired image using the perception algorithm in the microkernel operating system of the second processor 112, and the second recognition result is obtained by processing the first recognition result using the target operating system, and the target operating system is the macrokernel operating system of the second processor or the macrokernel operating system of the first processor.

[0037] Here, the first processor and the second processor may be any appropriate processors. During implementation, the first processor and the second processor may be the same or different.

[0038] The output device can be any suitable device capable of performing an output function, such as a display screen, an amplifier device, etc. For example, the target recognition result can be displayed on a display screen. Another example is the target recognition result can be played through the amplifier device.

[0039] A microkernel operating system is an operating system based on a microkernel architecture. The microkernel is a streamlined version of the kernel that provides the core functions of the operating system. It is designed to increase portability in a very small memory space and provide a modular design so that users can install different interfaces. Microkernel operating systems may include but are not limited to Windows NT, Unix, QNX, etc. The QNX system is a Unix-like real-time operating system that complies with the POSIX (Portable Operating System Interface of UNIX) specification. In some embodiments, the microkernel operating system of the first processor and the microkernel operating system of the second processor may be the same or different. For example, the microkernel operating system of the first processor and the microkernel operating system of the second processor are both QNX. For another example, the microkernel operating system of the first processor is QNX, and the microkernel operating system of the second processor is Unix.

[0040] A macro kernel operating system is an operating system based on a macro kernel architecture. The macro kernel includes all the functions required of a complete kernel. Macro kernel operating systems may include, but are not limited to, Linux, Android, etc. In some embodiments, the macro kernel operating system of the first processor and the macro kernel operating system of the second processor may be the same or different. For example, the macro kernel operating system of the first processor and the micro kernel operating system of the second processor may both be Android. For another example, the macro kernel operating system of the first processor may be Android, and the macro kernel operating system of the second processor may be Linux.

[0041] The perception algorithm may include but is not limited to the DMS perception algorithm, the OMS perception algorithm, the multi-modal perception algorithm, etc. The DMS perception algorithm is used to determine the state of the driving object. The state of the driving object may include but is not limited to whether it is tired, whether it is driving dangerously, whether it is driving passionately, whether it is smoking, etc. The OMS perception algorithm is used to determine the state of the passenger. The state of the passenger may include but is not limited to facial expressions, whether there are any left behind, etc. The multi-modal perception algorithm is used to fuse the DMS perception algorithm and the first information to determine the state of the driving object, or the OMS perception algorithm and the first information to determine the state of the passenger. Among them, the first information may include but is not limited to scene data, vehicle control information, ADS (Adaptive amping System, adjustable shock absorber system), etc. Vehicle control information may include but is not limited to vehicle speed information, braking information, or detection data of other sensors, etc. During implementation, the accuracy of the state of the driving object and the state of the passenger can be improved through the multi-modal perception algorithm.

[0042] The image acquisition method may include but is not limited to images acquired by an image acquisition device, images sent by other electronic devices, images pre-stored on the vehicle side, etc.

[0043] In some embodiments, the system further includes an image acquisition device communicatively connected to the second processor 112, and the second processor 112 is further configured to acquire the image captured by the image acquisition device. The image acquisition device includes at least one camera, and the image acquisition device includes a camera facing inside the vehicle and / or a camera facing outside the vehicle. For example, the image acquisition device includes a DMS camera, an OMS camera, a front-view camera, a surround-view camera, and the like. The videos captured by the front-view camera and the surround-view camera are mainly used for collision warning, lane warning, signal light / traffic sign recognition, and the like. There can be at least one DMS camera, and the DMS camera can be installed facing the driving area inside the vehicle to capture a video stream of the driving object. The video stream captured by the DMS camera can be used to identify the identity, gestures, status, and the like of the driving object. There can be at least one OMS camera, and the OMS camera can be installed facing the passenger area (i.e., non-driving area) inside the vehicle to capture a video stream of the passenger. The video stream captured by the OMS camera can be used to identify the gestures, status, and the like of the passenger.

[0044] The first recognition result is the recognition result corresponding to the perception algorithm. The first recognition results corresponding to different perception algorithms can be the same or different. During implementation, the output methods of different first recognition results can be the same or different. For example, when the multimodal perception algorithm detects that the driver is in a state of fatigue (corresponding to the aforementioned first recognition result), the output can include using the power amplifier device to play and / or displaying the corresponding prompt information on the display screen. The differentiated display may include, but is not limited to, highlighting, dimming, etc. The highlighted display may include, but is not limited to, pop-up windows, highlighting, framing, color change, bolding, changing font size, adding special marks, adding shadows, etc. The dimmed display may include, but is not limited to, darkening, changing color, graying, thinning, etc. For example, the power amplifier device may play "Please drive to a safe area to rest" and / or a pop-up window may be displayed on the display screen. In this way, by displaying the corresponding prompt information through the power amplifier device, the driver and / or passengers can handle it in a timely manner, thereby improving the user experience.

[0045] For another example, when the multimodal sensing algorithm detects that there is something left in the passenger area (i.e., the front passenger seat and / or the rear seating area) (corresponding to the aforementioned first recognition result), the output may include playing the corresponding prompt information through the power amplifier device and / or displaying it on the display screen.

[0046] The second recognition result is obtained by processing the first recognition result. In some embodiments, the macro kernel operating system includes a multi-mode fusion algorithm, and the second recognition result is the result corresponding to the multi-mode fusion algorithm. During implementation, the multi-mode fusion algorithm is used to fuse the first recognition result and the second information to determine the state of the driving object and / or the passenger. The second information may include but is not limited to instrument application data, user data, voice service data, etc. User data may include but is not limited to voice history data, visual history data, etc. For example, the second recognition result may include the state of the driving object determined based on the DMS perception algorithm + vehicle control information + user data. For another example, the second recognition result may include the state of the passenger determined based on the OMS perception algorithm + user data.

[0047] In some embodiments, the microkernel operating system of the second processor 112 is communicatively connected to the macrokernel operating system of the second processor 102 , the microkernel operating system of the first processor 111 , and / or the macrokernel operating system of the first processor 111 .

[0048] Here, the microkernel operating system of the second processor 112 communicates with other operating systems to pass the output results of the multimodal perception algorithm to the macrokernel operating system for processing, or obtain the first information from other microkernel operating systems or macrokernel operating systems.

[0049] In some embodiments, the vehicle's instrument application is deployed in the microkernel operating system of the first processor, the vehicle's first business application is deployed in the macrokernel operating system of the first processor, and the vehicle's second business application is deployed in the macrokernel operating system of the second processor, and the type of the first business application is different from the type of the second business application.

[0050] Here, since instrument applications also require high real-time performance and stability, they can be deployed in a microkernel operating system. Since microkernel operating systems have less memory, business applications need to be deployed in a macrokernel operating system.

[0051] Application services can be divided into processing procedures, business performance, etc. according to the type of business. Among them, the processing procedure may include but is not limited to scenario applications (used to describe at least one usage scenario), ADS, vehicle control applications, voice services (used to provide voice-to-text, text-to-speech and other functions through voice resources), audio services (used to provide audio playback capabilities for vehicles), etc. Business performance may include but is not limited to multi-mode fusion algorithms, display applications, voice semantic algorithms, voice engines (referring to voice synthesis engines, used to convert other information into audible sound information), etc. During implementation, the second business of the vehicle can also be deployed in the macro-kernel operating system of the first processor, and the first business of the vehicle can be deployed in the macro-kernel operating system of the second processor.

[0052] In some embodiments, the system also includes communication middleware, which is used for at least one of the following: establishing a communication connection between the microkernel operating system of the second processor and the macrokernel operating system of the second processor; establishing a communication connection between the microkernel operating system of the second processor and the microkernel operating system of the first processor; establishing a communication connection between the microkernel operating system of the second processor and the macrokernel operating system of the first processor.

[0053] Here, the communication middleware can be any suitable component capable of implementing communication functions, such as SOMEIP (Scalable Service-Oriented Middleware over Internet Protocol) and VSOMEIP. SOMEIP is a cross-platform communication protocol that publishes information only when the recipient requests it (i.e., a subscribe / publish mechanism). This service-oriented data communication method allows for significant changes to the vehicle's hardware without requiring changes to the communication software layer. VSOMEIP is an open-source SOMEIP library that primarily implements SOMEIP's communication and service discovery capabilities, adding security mechanisms. VSOMEIP not only enables communication between two devices within SOMEIP (i.e., TCP and UDP), but also enables cross-process communication within the same device (i.e., Domain Sockets). By using VSOMEIP as cross-device and cross-system communication middleware, different businesses can customize messages based on their needs without requiring real-time message requests, reducing bandwidth usage and improving the real-time and reliability of message delivery.

[0054] Figure 1B is a schematic diagram of the composition structure of a vehicle cabin control system provided by an embodiment of the present disclosure. As shown in Figure 1B, the vehicle cabin control system 10 includes a control device 11 and an output device 12, wherein the control device 11 includes at least a first processor 111 and a second processor 112 communicating with the first processor 111. Through the communication middleware of the cabin control system, communication connections are established between the microkernel operating system in the second processor 112 and the macrokernel operating system in the second processor 112, the microkernel operating system of the first processor 111, and the macrokernel operating system of the first processor 111.

[0055] In some implementations, the macro kernel operating system of the second processor is further configured to: obtain the first recognition result using a preset first communication method.

[0056] Here, the first communication mode may be any appropriate mode, such as subscription, request, etc. During implementation, the macro kernel operating system of the second processor may obtain the first identification result from the micro kernel operating system of the second processor through VSOMEIP, that is, obtain the first identification result through a subscription / publishing mechanism.

[0057] In some implementations, the macro kernel operating system of the first processor is further configured to: obtain the first recognition result using the first communication method.

[0058] Here, the first communication mode may be any appropriate mode, such as subscription, request, etc. During implementation, the macro kernel operating system of the first processor may obtain the first identification result from the micro kernel operating system of the second processor through VSOMEIP, that is, obtain the first identification result through a subscription / publishing mechanism.

[0059] In some embodiments, the microkernel operating system of the first processor is further configured to: obtain the first recognition result using a preset second communication method; wherein the first communication method includes at least the second communication method.

[0060] Here, the second communication mode may be subscription. In implementation, the microkernel operating system of the first processor may obtain the first identification result from the microkernel operating system of the second processor via VSOMEIP, that is, obtain the first identification result via a subscription / publishing mechanism.

[0061] FIG1C is a schematic diagram of the structure of a control device provided by an embodiment of the present disclosure. As shown in FIG1C , the control device includes an A core 111 (corresponding to the aforementioned second processor) and a B core 112 (corresponding to the aforementioned first processor), wherein:

[0062] Core A includes a first QNX operating system (corresponding to the aforementioned microkernel operating system) and a first Android operating system (corresponding to the aforementioned macrokernel operating system), wherein an instrument application 113 is deployed in the first QNX operating system and a first business application 114 is deployed in the first Android operating system;

[0063] The B core includes a second QNX operating system (corresponding to the aforementioned microkernel operating system) and a second Android operating system (corresponding to the aforementioned macrokernel operating system). The perception algorithm service 115 is deployed in the second QNX operating system, and the second business application 116 is deployed in the second Android operating system.

[0064] During implementation, the instrument application 113 subscribes to the first recognition result from the perception algorithm 115 through VSOMEIP; the first business application 114 obtains the first recognition result from the instrument application 113 through subscription / request; the second business application 116 obtains the first recognition result from the instrument application 113 through subscription / request.

[0065] FIG1D is a schematic diagram of the structure of a cockpit control system provided by an embodiment of the present disclosure. As shown in FIG1D , the cockpit control system 10 includes a control device 11, an output device 12, and an image acquisition device 13, wherein:

[0066] The control device 11 includes a first processor 111 and a second processor 112. The first processor 111 includes a first QNX operating system (corresponding to the aforementioned microkernel operating system) and a first Android operating system (corresponding to the aforementioned macrokernel operating system). The second processor 112 includes a second QNX operating system (corresponding to the aforementioned microkernel operating system) and a second Android operating system (corresponding to the aforementioned macrokernel operating system).

[0067] (1) The instrument application 113 is deployed in the first QNX operating system, and the instrument application 113 is respectively communicated with the output device 12 and the multi-mode perception algorithm service 115 in the second QNX operating system;

[0068] (2) A first business application 114 is deployed in the first Android operating system. The first business application 114 may include but is not limited to scenario application, ADS service, vehicle control, voice service, audio service, user data, etc. During implementation, the scenario application, ADS service and vehicle control are communicated with the multi-mode perception algorithm service 115 in the second QNX operating system; the voice service and user data are communicated with the multi-mode fusion algorithm service in the second Android operating system; the voice service is also communicated with the voice engine in the second Android operating system; the audio service is communicated with the digital signal processor, which can be used to process the output of the audio service and the output of the voice engine in the second QNX operating system, and pass the processing results to the output device 12, and can also be used to process the input of the input device (for example, a microphone) and pass the processing results to the multi-mode fusion algorithm service in the second Android operating system;

[0069] (3) The perception algorithm service 115, the camera service 117, and the process management 118 are deployed in the second QNX operating system. The perception algorithm service 115 includes the OMS perception algorithm service, the DMS perception algorithm service, and the multi-mode perception algorithm service. The image acquisition device 13 transmits the acquired video stream to the camera service 117 or directly transmits it to the DMS / OMS display application in the second Android operating system for display. The camera service 117 pre-processes the video stream and sends the processed image to the DMS perception algorithm service and the OMS perception algorithm service respectively. The DMS perception algorithm service processes the image to obtain the first state of the driver and transmits the first state of the driver to the multi-mode perception algorithm service. The OMS perception algorithm service recognizes the image. The first state of the passenger is obtained by performing separate processing, and the first state of the passenger is transmitted to the multimodal perception algorithm service. The multimodal perception algorithm service updates the first state of the driver and the first state of the passenger based on the first information, obtains the second state of the driver (corresponding to the aforementioned first recognition result) and the second state of the passenger (corresponding to the aforementioned first recognition result), and transmits the second state of the driver and the second state of the passenger to the multimodal fusion algorithm service and / or the DMS / OMS display application (i.e., in a subscription manner); the process management 118 is mainly used to manage system memory, operating system startup, sleep, initialization, etc. Since the process management is highly correlated with the perception algorithm and does not occupy much memory, the process management 118 can be deployed in the second QNX operating system;

[0070] (4) A second business application 116 is deployed in the second Android operating system. The second business application 116 may include but is not limited to a multi-mode fusion algorithm service, a DMS / OMS display application, a speech semantic algorithm service, a speech engine, etc. The multi-mode fusion algorithm service updates the second state of the driver and the second state of the passenger based on the second information, obtains the third state of the driver (corresponding to the aforementioned second recognition result) and the third state of the passenger (corresponding to the aforementioned second recognition result), and transmits the third state of the driver and the third state of the passenger to the DMS / OMS display application. The DMS / OMS display application is communicated with the output device 12, and the speech semantic algorithm service is communicated with the speech engine. The speech engine is used to process the output of the multi-mode fusion algorithm service and the output of the speech service in the first Android operating system, and transmit the processing result to the output device 12.

[0071] During implementation, the second Android operating system, the first Android operating system, and the first QNX operating system can all obtain the first recognition result from the second QNX operating system by subscription.

[0072] In some embodiments, when the output device 12 is connected to a voice engine or audio service, a sound system is provided to implement voice interaction or audio playback functions.

[0073] In some embodiments of the present disclosure, the sound system can simultaneously support both in-vehicle and out-of-vehicle sound modes, which requires higher on-board computing power than the traditional single in-vehicle sound scenario. The cockpit control system provided by the present disclosure utilizes dual cores to provide high computing power, accepts user voice commands through the voice service deployed in core A (corresponding to the aforementioned first processor 111), obtains user status information through the multimodal perception algorithm service deployed in core B (corresponding to the aforementioned second processor 112), and through the multimodal fusion algorithm service, speech semantic algorithm service, and voice engine deployed in core B, can quickly and accurately identify user intent and conduct voice interaction with the user, and can be applied to both in-vehicle and out-of-vehicle sound scenarios.

[0074] To achieve both in-vehicle and out-of-vehicle sound generation modes, in some embodiments of the present disclosure, the sound generation system includes a panel sound generation module at the hardware level. The panel sound generation module can be a piezoelectric sound generation module, an electromagnetic induction sound generation module, or a traditional speaker. In a specific example, the piezoelectric sound generation module can include a piezoelectric ceramic speaker. The piezoelectric ceramic speaker can include an electrode sheet configured to receive an excitation voltage from a drive circuit. The electrode sheet can be a pair of positive and negative electrode sheets. The piezoelectric ceramic speaker can also include a piezoelectric ceramic configured to expand or contract laterally or longitudinally in response to the excitation voltage received via the electrode sheet. As previously described, the excitation voltage can be a high-frequency square wave with alternating polarity, so that the piezoelectric ceramic will undergo mechanical deformation, i.e., expansion or contraction, in response to the alternating polarity square wave. Furthermore, the piezoelectric ceramic can be transversely or longitudinally polarized, thereby producing transverse or longitudinal mechanical deformation in response to the excitation voltage. The piezoelectric ceramic speaker can also include a vibration plate that is bonded to the piezoelectric ceramic and vibrates as the piezoelectric ceramic expands or contracts. In this way, the piezoelectric ceramic speaker converts the input excitation voltage into vibrations, thereby emitting sound. The vibration plate can be attached to a component such as a door panel to drive the component to vibrate. The larger the area of ​​the component such as the door panel driven by the piezoelectric ceramic speaker, the stronger the low-frequency response of the vibration and the weaker the high-frequency response. The piezoelectric ceramic speaker may also include a vibration gasket located between the vibration plate and the component such as the door panel. The vibration gasket can be used to adjust the frequency response of the vibration generated by the vibration plate. For example, if the low-frequency response of the vibration needs to be enhanced, the area of ​​the vibration gasket can be increased; if the low-frequency response needs to be weakened, the area of ​​the vibration gasket can be reduced. The panel sound module enhances the vehicle's sound field. In the examples disclosed herein, the panel sound module can be installed in at least one of the vehicle's door interior panel, armrest, seat headrest, hood, trunk lid, roof, chassis, rearview mirror, front bumper, or rear bumper.

[0075] In the disclosed embodiment, on the one hand, by deploying a dual-core control device in the cockpit control system and deploying dual systems on each processor, and deploying the perception algorithm in the microkernel operating system of a certain processor, the algorithm reasoning and the business are made independent of each other. Then, other businesses can be executed normally while executing algorithm reasoning, thereby improving the execution efficiency of the business. On the other hand, using VSOMEIP as a cross-device and cross-system communication middleware allows different businesses to customize messages according to needs without the need for real-time message requests, thereby reducing bandwidth occupancy and improving the real-time and reliability of message transmission.

[0076] Based on the above embodiments, the embodiments of the present disclosure further provide an image processing method, which can be executed by an electronic device. The electronic device can be various types of terminals such as laptops, tablet computers, desktop computers, and vehicle terminals, and can also be implemented as a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. Figure 2 is a schematic diagram of the implementation flow of an image processing method provided by an embodiment of the present disclosure, which is applied to the cockpit control system of any of the vehicles described above. As shown in Figure 2, the method includes steps S21 to S22, wherein:

[0077] Step S21: Use the perception algorithm in the microkernel operating system of the second processor to recognize the acquired image and obtain a first recognition result.

[0078] Here, the perception algorithm may include, but is not limited to, a DMS perception algorithm, an OMS perception algorithm, a multimodal perception algorithm, and the like. The first recognition result corresponds to the perception algorithm. The first recognition results corresponding to different perception algorithms may be the same or different. For example, a DMS perception algorithm and a multimodal perception algorithm may output the state of the driving object, while an OMS perception algorithm and a multimodal perception algorithm may output the state of the passenger.

[0079] Step S22, output the first recognition result and / or the second recognition result; wherein, the second recognition result is obtained by processing the first recognition result using the target operating system, and the target operating system is the macro kernel operating system of the second processor or the macro kernel operating system of the first processor.

[0080] Here, the output method may include but is not limited to displaying the corresponding recognition result using a display screen, playing the corresponding recognition result using an amplifier device, etc. The output methods of different first recognition results may be the same or different. The second recognition result is obtained by processing the first recognition result. In some embodiments, the macro kernel operating system includes a multi-mode fusion algorithm, and the second recognition result is the result corresponding to the multi-mode fusion algorithm. During implementation, the multi-mode fusion algorithm is used to fuse the first recognition result and the second information to determine the status of the driving object and / or passenger. Among them, the second information may include but is not limited to instrument application data, user data, voice service data, etc.

[0081] In an embodiment of the present disclosure, the acquired image is recognized by utilizing the perception algorithm in the microkernel operating system of the second processor to obtain a first recognition result, and the first recognition result and / or the second recognition result are output; wherein the second recognition result is obtained by processing the first recognition result using a target operating system, and the target operating system is the macrokernel operating system of the second processor or the macrokernel operating system of the first processor. In this way, by deploying the perception algorithm in the microkernel operating system of the second processor, the algorithm reasoning and the business are made independent of each other, so that other businesses can be executed normally while executing the algorithm reasoning, thereby improving the execution efficiency of the business.

[0082] In some embodiments, the method further comprises step S23, wherein:

[0083] Step S23: The target operating system uses a preset first communication method to obtain the first recognition result, and uses a preset multi-modal fusion algorithm to fuse the first recognition result with at least one modality data to obtain the second recognition result.

[0084] Here, the first communication mode may be any appropriate mode, such as subscription, request, etc. The at least one modality data may include but is not limited to meter application data, user data, voice service data, etc.

[0085] The second recognition result may include but is not limited to the state of the driving object, the state of the passenger, etc.

[0086] In the disclosed embodiment, on the one hand, the first recognition result is obtained by subscription, which reduces bandwidth usage and improves the real-time and reliability of message transmission; on the other hand, the status of the driving object / passenger is obtained through a multi-modal fusion algorithm, which improves the accuracy of the status of the driving object / passenger, thereby improving the user experience.

[0087] In some embodiments, the cockpit control system further includes an image acquisition device communicatively connected to the second processor, and the method further includes step S24, wherein:

[0088] Step S24: The second processor obtains the image captured by the image capture device.

[0089] In some embodiments, the method further comprises step S25, wherein:

[0090] Step S25: The microkernel operating system of the first processor obtains the first recognition result using a preset second communication method.

[0091] The description of the above method embodiment is similar to the description of the above system embodiment, and has similar beneficial effects as the system embodiment. For technical details not disclosed in the method embodiment of the present disclosure, please refer to the description of the system embodiment of the present disclosure for understanding.

[0092] Based on the above embodiments, an embodiment of the present disclosure provides an image processing device for use in a vehicle cockpit control system. FIG3 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure. As shown in FIG3 , the image processing device 30 includes a recognition portion 31 and an output portion 32, wherein:

[0093] The recognition part 31 is configured to recognize the acquired image using a perception algorithm in the microkernel operating system of the second processor to obtain a first recognition result;

[0094] The output part 32 is configured to output the first recognition result and / or the second recognition result; wherein, the second recognition result is obtained by processing the first recognition result using the target operating system, and the target operating system is the macro kernel operating system of the second processor or the macro kernel operating system of the first processor.

[0095] In some embodiments, the device also includes a determination part in the target operating system, and the determination part is configured to use a preset first communication method to obtain the first recognition result, and use a preset multimodal fusion algorithm to fuse the first recognition result with at least one modality data to obtain the second recognition result.

[0096] In some embodiments, the cockpit control system further includes an image acquisition device communicatively connected to the second processor, and the apparatus further includes a first acquisition part in the second processor, wherein the first acquisition part is configured to acquire the image acquired by the image acquisition device.

[0097] In some embodiments, the apparatus further includes a second acquisition part in the first processor, and the second acquisition part is configured to acquire the first recognition result by using a preset second communication method.

[0098] In some embodiments, the first recognition result includes at least one of the following: a state of the driving object, a state of a passenger.

[0099] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0100] In the embodiments of the present disclosure and other embodiments, "part" can be part of a circuit, part of a processor, part of a program or software, etc., and of course it can also be a unit, a module, or a non-modular one.

[0101] It should be noted that, in the embodiments of the present disclosure, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific combination of hardware and software.

[0102] An embodiment of the present disclosure provides a vehicle, comprising any of the above-mentioned cabin control systems.

[0103] An embodiment of the present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above method when executing the computer program.

[0104] The present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the above method when executed by a processor. The computer-readable storage medium may be transient or non-transient.

[0105] The present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0106] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

[0107] It should be understood that “one embodiment” or “an embodiment” mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, “in one embodiment” or “in an embodiment” appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0108] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0109] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0110] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0111] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0112] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0113] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0114] The above description is only an embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, and they should all be included in the protection scope of the present disclosure. Industrial Applicability

[0115] The present disclosure provides a vehicle cockpit control system, an image processing method and apparatus, a vehicle, an electronic device, a storage medium, and a computer program product. The vehicle cockpit control system includes a control device and an output device communicatively connected to the control device. The control device includes at least a first processor and a second processor communicating with the first processor, the first processor and the second processor both including a microkernel operating system and a macrokernel operating system. The output device is configured to output a target recognition result. The target recognition result includes a first recognition result and / or a second recognition result. The first recognition result is obtained by processing a captured image using a perception algorithm in the microkernel operating system of the second processor, and the second recognition result is obtained by processing the first recognition result using a target operating system, the target operating system being the macrokernel operating system of the second processor or the macrokernel operating system of the first processor. The above solution achieves independence between algorithm reasoning and business operations by deploying a dual-core control device in the cockpit control system, deploying dual systems on each processor, and deploying the perception algorithm in the microkernel operating system of a processor. This allows other business operations to be executed normally while the algorithm reasoning is executed, thereby improving business execution efficiency.

Claims

1. A cockpit control system for a vehicle, comprising a control device and an output device connected to the control device for communication, wherein the control device comprises at least a first processor and a second processor communicating with the first processor, wherein the first processor and the second processor both comprise a microkernel operating system and a macrokernel operating system; wherein: The output device is used to output the target recognition result; wherein the target recognition result includes a first recognition result and / or a second recognition result, the first recognition result is obtained by processing the acquired image using the perception algorithm in the microkernel operating system of the second processor, and the second recognition result is obtained by processing the first recognition result using a target operating system, and the target operating system is the macrokernel operating system of the second processor or the macrokernel operating system of the first processor.

2. The system according to claim 1, wherein: The microkernel operating system of the second processor is communicatively connected with the macrokernel operating system of the second processor, the microkernel operating system of the first processor, and / or the macrokernel operating system of the first processor.

3. The system according to claim 1 or 2, wherein: The system further includes a communication middleware, wherein the communication middleware is used for at least one of the following: Establishing a communication connection between the microkernel operating system of the second processor and the macrokernel operating system of the second processor; Establishing a communication connection between the microkernel operating system of the second processor and the microkernel operating system of the first processor; A communication connection is established between the microkernel operating system of the second processor and the macrokernel operating system of the first processor.

4. The system according to claim 3, wherein: The communication middleware includes VSOMEIP.

5. The system according to claim 1 or 2, wherein: The macro kernel operating system of the second processor is further used to: obtain the first recognition result by using a preset first communication method; The macro kernel operating system of the first processor is further used to: obtain Taking the first recognition result; The microkernel operating system of the first processor is further used to: obtain the first recognition result by using a preset second communication method; wherein the first communication method at least includes the second communication method.

6. The system according to claim 1 or 2, wherein: The system also includes an image acquisition device communicatively connected to the second processor; The second processor is further used to obtain the image captured by the image acquisition device.

7. The system according to claim 1 or 2, wherein: The vehicle's instrument application is deployed in the microkernel operating system of the first processor, the vehicle's first business application is deployed in the macrokernel operating system of the first processor, and the vehicle's second business application is deployed in the macrokernel operating system of the second processor. The type of the first business application is different from the type of the second business application.

8. An image processing method, applied to a cockpit control system of a vehicle, the cockpit control system comprising a control device, the control device comprising a first processor and a second processor communicating with the first processor, the first processor and the second processor both comprising a microkernel operating system and a macrokernel operating system, the method comprising: Using a perception algorithm in a microkernel operating system of the second processor, the acquired image is recognized to obtain a first recognition result; Output the first recognition result and / or the second recognition result; wherein, the second recognition result is obtained by processing the first recognition result using a target operating system, and the target operating system is the macro kernel operating system of the second processor or the macro kernel operating system of the first processor.

9. The method according to claim 8, wherein: The method further comprises: The target operating system uses a preset first communication method to obtain the first recognition result, and uses a preset multi-modal fusion algorithm to fuse the first recognition result with at least one modality data to obtain the second recognition result.

10. The method according to claim 8 or 9, wherein: The first recognition result includes at least one of the following: a state of the driving object and a state of the passenger.

11. An image processing device, applied to a cockpit control system of a vehicle, the cockpit control system comprising a control device, the control device comprising a first processor and a second processor communicating with the first processor, the first processor and the second processor both comprising a microkernel operating system and a macrokernel operating system, the device comprising: The recognition part is configured to recognize the acquired image by using the perception algorithm in the microkernel operating system of the second processor to obtain a first recognition result; The output part is configured to output the first recognition result and / or the second recognition result; wherein, the second recognition result is obtained by processing the first recognition result using a target operating system, and the target operating system is the macro kernel operating system of the second processor or the macro kernel operating system of the first processor.

12. The device according to claim 11, wherein: The apparatus also includes a determination portion located in the target operating system; The determination part is configured to obtain the first recognition result by using a preset first communication method, and to fuse the first recognition result with at least one modality data by using a preset multi-modal fusion algorithm to obtain the second recognition result.

13. A vehicle comprising the cockpit control system according to any one of claims 1 to 7.

14. An electronic device comprising a processor and a memory, wherein the memory stores a computer program executable on the processor, and the processor implements the method according to any one of claims 8 to 10 when executing the computer program.

15. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 8 to 10 when executed by a processor. 16 . A computer program product, comprising a computer program or instructions, which, when the computer program or instructions are run on an electronic device, causes the electronic device to execute the method according to claim 8 .

Images