Automobile cockpit monitoring method, device, related equipment and computer program product

Abstract

The present application discloses a vehicle cabin monitoring method, device, related equipment and computer program product, which relates to the field of vehicle cabin monitoring technology. The method includes: obtaining multimodal information monitored from the vehicle cabin environment; obtaining prompt instructions corresponding to each monitoring task in a multitask prompt instruction set according to various types of pre-configured monitoring tasks; using multimodal information and prompt instructions of each monitoring task to generate monitoring results of each monitoring task. The present application can integrate different monitoring tasks in the vehicle cabin, and different monitoring tasks can share multimodal information, giving full play to the complementary effects of each modal information, optimizing the complementarity between different monitoring tasks, and improving the overall monitoring performance. In addition, the present application can realize cross-task processing through a unified framework, without the need to deploy each independent task function module separately, thereby reducing the burden of deploying related functional models.

Description

Technical Field

[0001] The present application relates to the technical field of automobile cockpit monitoring, and more specifically, to an automobile cockpit monitoring method, apparatus, related equipment and computer program product. Background Art

[0002] With the continuous development of deep learning, the smart cockpit of automobiles has gradually transitioned from a mechanical cockpit with basic control functions to a smart cockpit with software and hardware upgrades and diversified interaction methods.

[0003] In the field of modern automobiles, the intelligent cockpit system integrates a series of functions, such as voice recognition tasks, fatigue warning tasks, gesture control tasks, gaze tracking tasks, emotion recognition tasks, etc., making human-computer interaction more convenient. However, each task is currently designed and deployed based on independent capabilities, which means that whether it is from data construction, structure setting, training reasoning process, or final model deployment, each task is a relatively independent system. Although this modular development approach has its advantages, it also brings significant challenges to the overall development and coordination of the system. The independent operation of each task not only limits the potential synergy, but may also encounter bottlenecks in performance improvement and add additional burdens when deploying the model. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a vehicle cockpit monitoring method, device, related equipment and computer program product to solve at least one defect existing when each task of the traditional vehicle cockpit system is independently developed and deployed. The specific solution is as follows:

[0005] In a first aspect, a vehicle cabin monitoring method is provided, comprising:

[0006] Obtain multimodal information from monitoring the vehicle cabin environment;

[0007] According to the pre-configured monitoring tasks of various types, a prompt instruction corresponding to each of the monitoring tasks is obtained in the configured multi-task prompt instruction set, wherein the prompt instruction corresponding to any monitoring task includes query description information of the monitoring result or related information of the monitoring task;

[0008] The monitoring result of each monitoring task is generated by using the multimodal information and the prompt instruction of each monitoring task.

[0009] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0010] Based on the monitoring result of each monitoring task, a response operation matching the monitoring result is executed according to the configured control strategy.

[0011] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0012] Based on the monitoring results of the preset target type monitoring tasks, identify the interaction intention of the user in the car cabin;

[0013] A response to the user's interaction intention is output.

[0014] In a possible design, in another implementation of the first aspect of the embodiments of the present application, before outputting a response to the user's interaction intention, the method further includes:

[0015] Identify the identity information of the user who currently has the intention to interact;

[0016] The process of outputting a response to the user's interaction intention includes:

[0017] A response matching the interaction intention and identity information of the user is determined, and the response is output.

[0018] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of generating a monitoring result of each monitoring task by using the multimodal information and the prompt instruction of each monitoring task includes:

[0019] Sending the multimodal information and the prompt instruction of each monitoring task into the configured multi-task processing model to obtain the monitoring result of each monitoring task output by the model;

[0020] Among them, the multi-task processing model is configured as follows: feature extraction and alignment are performed on each modal information in the input multimodal information and the prompt instruction of each monitoring task to obtain the aligned modal features and prompt instruction features; encoding and decoding operations are performed on the aligned modal features and the prompt instruction features to obtain the monitoring results of each monitoring task.

[0021] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the training process of the multi-task processing model includes:

[0022] Acquire multimodal training samples monitored for a car cabin scene, question description samples for the multimodal training samples, and matching answer text labels;

[0023] Inputting the multimodal training samples and the problem description samples into a multi-task processing model to obtain an output result of the model;

[0024] A loss function value is calculated based on the output result and the answer text label, and the parameters of the multi-task processing model are updated according to the loss function value until the set model training end condition is reached.

[0025] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the multimodal training sample includes an image training sample, and the problem description sample includes at least one of the following:

[0026] A first problem description sample is requested to perform a global description on the image training sample;

[0027] A second problem description sample requesting a local description of the image training sample;

[0028] a third question description sample for asking questions about the visual content of the image training sample;

[0029] The answer text label matching the first question description sample is a global description text of the image training sample;

[0030] The answer text label matching the second question description sample is a local description text of the image training sample;

[0031] The answer text label that matches the third question description sample is the answer text corresponding to the question.

[0032] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the multimodal information includes information of at least two or more of the following modalities:

[0033] Information from audio, video, image, depth and physiological signal modalities.

[0034] In a second aspect, a vehicle cabin monitoring device is provided, comprising:

[0035] A multimodal information acquisition unit, used to acquire multimodal information monitored from the vehicle cabin environment;

[0036] A prompt instruction acquisition unit, used to acquire a prompt instruction corresponding to each of the pre-configured types of monitoring tasks in the configured multi-task prompt instruction set, wherein the prompt instruction corresponding to any monitoring task includes inquiry description information of the monitoring result or related information of the monitoring task;

[0037] The monitoring result generating unit is used to generate the monitoring result of each monitoring task by using the multimodal information and the prompt instruction of each monitoring task.

[0038] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0039] The memory is used to store programs;

[0040] The processor is used to execute the program to implement the vehicle cabin monitoring method described in any one of the first aspects of the present application.

[0041] In a fourth aspect, a car is provided, comprising the electronic device described in the third aspect.

[0042] In a fifth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the vehicle cabin monitoring method described in any one of the first aspects of the present application is implemented.

[0043] In a sixth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the vehicle cabin monitoring method described in any one of the aforementioned first aspects of the present application.

[0044] By means of the above technical solution, the present application pre-arranges various types of monitoring tasks that need to be implemented in the car cockpit scene, such as voice recognition tasks, fatigue warning tasks, gesture control tasks, sight tracking tasks, emotion recognition tasks, etc. At the same time, a multi-task prompt instruction set is pre-configured, which contains multiple preset prompt instructions. The prompt instruction corresponds to the monitoring task, which may include inquiry description information of the monitoring result or related information of the monitoring task. Taking the voice recognition task as an example, the corresponding prompt instruction may include "Is there anyone talking in the car at present? What is the content of the conversation?" etc. On this basis, the multi-modal information monitored for the car cockpit environment is obtained. At the same time, for each type of monitoring task, the corresponding prompt instruction is obtained from the multi-task prompt instruction set, and the monitoring result of each monitoring task is predicted and generated by using the simultaneous instruction and the obtained multi-modal information. Obviously, this application provides a unified framework that can integrate different monitoring tasks in the car cockpit. When generating the monitoring results of each monitoring task, the acquired multimodal information and the prompt instructions corresponding to the monitoring task can be used. Different monitoring tasks can share multimodal information, give full play to the complementary role of each modal information, optimize the complementarity between different monitoring tasks, and thus improve the overall monitoring performance. In addition, this application can realize cross-task processing through a unified framework, without the need to deploy each independent task function module separately, thereby reducing the burden of deploying related functional models. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0046] Figure 1 A schematic diagram of a system architecture for implementing the vehicle cabin monitoring method provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of various functional tasks that can be realized by an automobile intelligent cockpit provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a process flow of a vehicle cabin monitoring method provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the overall framework of a vehicle cabin monitoring method provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of image-text pair training data provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of a vehicle cabin monitoring device provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0055] The present application provides a vehicle cabin monitoring method, which can be applied to Figure 1 The system architecture shown may include a vehicle-mounted terminal, sensors, and a server (optional).

[0056] The vehicle terminal may be a vehicle controller, which is responsible for the coordination of various tasks, data calculation and processing, etc. In this embodiment, the vehicle terminal may be used to implement various types of monitoring tasks for the vehicle cockpit and intelligently interact with the user.

[0057] Sensors may include various types, for example, microphone arrays, cameras, depth sensors, olfactory sensors, tactile sensors, physiological sensors, inertial sensors, etc. Taking cameras as an example, they can collect images of the interior of the cabin, such as images of the drivers and passengers in the cabin, etc. Physiological sensors include but are not limited to: heart rate monitors, skin electrical sensors, etc., which can collect the user's physiological data.

[0058] The server can be a remote device connected to the vehicle terminal. In this application, the server is optional. When the vehicle cockpit monitoring system does not include a server, the vehicle cockpit monitoring method can be implemented specifically through the vehicle terminal. When the vehicle cockpit monitoring system includes a server, the server can cooperate with the vehicle terminal to complete the vehicle cockpit monitoring system.

[0059] In the modern automotive field, the smart cockpit system integrates a series of biometric technologies, including fingerprint recognition, iris or eye tracking, and heart rate monitoring. These features not only enhance safety, but also enhance the personalized experience. In terms of auditory interaction, the system achieves seamless communication with the driver and passengers through voice recognition, voiceprint recognition, and personalized sound settings. Visual interaction covers functions such as facial recognition unlocking, gesture control, eye tracking, and emotion recognition, making the interaction more intuitive and humane. Finally, natural language processing and human-computer interaction technologies such as language understanding, knowledge question and answer, and multi-round dialogue capabilities provide an interactive and conversational environment for the smart cockpit, greatly improving the level of intelligence in driving and riding.

[0060] The diverse tasks in current automotive smart cockpits show significant diversity in input and output types, perception granularity, and modeling methods. Examples include:

[0061] For visual interaction tasks, face recognition and gaze perception usually perform spatial analysis based on single-frame image input and output relevant classification and attribute information.

[0062] For gesture recognition and emotion detection tasks, which rely on video streams, feature extraction and analysis are also required in the time dimension.

[0063] For auditory interaction tasks, such as speech recognition and voiceprint recognition, different signal processing models, acoustic models, language models and decoding strategies are required, and the output converts the sound into text information or voiceprint attributes respectively.

[0064] The core of natural language understanding tasks is to convert the natural language text provided by users into structured information that machines can understand and process. This process includes not only identifying the user's intention, that is, what operation they want to complete or what kind of information they want to get, but also extracting key entities from the user's input, such as time, place, object and other important parameters and reference information.

[0065] For human-computer interaction tasks, it is a broader concept, which not only includes the ability of natural language understanding tasks, but also covers how the system generates appropriate responses and performs corresponding actions based on the results of understanding. In addition, advanced human-computer interaction may also include dialogue management, so that the system can have multiple rounds of dialogue with users, resolve ambiguous instructions, and even predict user needs in advance to provide more intelligent and humane services.

[0066] The current intelligent cockpit of automobiles designs and deploys models for each functional task based on independent atomic capabilities, and each task is a relatively independent system. Since multiple modal information works independently, the complementary effects between the modalities cannot be brought into play, and unified and coordinated multimodal perception cannot be truly achieved. In addition, deploying the processing model for each task separately will also increase the burden of model deployment.

[0067] In response to the above problems, this application aims to propose a unified modeling framework that can integrate different functional tasks in the smart cockpit and build a new monitoring system for car cockpit passengers, which can realize the monitoring and interaction of passengers. Through the method of this application, efficient cross-task synergy can be achieved, the complementarity between different functional tasks can be optimized, and the time and cost of data preparation and model deployment can be reduced, moving from single-modal perception interaction or the current mainstream series connection of various modal functions to true multi-modal information fusion and perception interaction.

[0068] Combination Figure 2 As shown, the method of this application can integrate multiple modal information, such as audio, biological, image, depth and other modal information. On this basis, it integrates different functional tasks in the smart cockpit to achieve monitoring and intelligent interaction of drivers and passengers. Figure 2 As shown, the functional tasks include but are not limited to: speech recognition, face recognition, gesture recognition, emotion perception, fatigue perception, distraction perception, gaze perception, behavior detection, depth estimation, health monitoring, language understanding, knowledge question answering, etc.

[0069] Next, the vehicle cabin monitoring method provided by the present application is introduced in conjunction with specific embodiments. Figure 3 The vehicle cabin monitoring method specifically comprises the following steps:

[0070] Step S100: Acquire multimodal information monitored from the vehicle cabin environment.

[0071] Specifically, the present application can set up various types of sensors in the car cabin environment to collect multimodal information, including but not limited to microphone arrays, cameras, depth sensors, olfactory sensors, tactile sensors, physiological sensors, inertial sensors, etc. Examples of collected multimodal information include: audio, video, image, depth signal, physiological signal modality information, etc.

[0072] By acquiring multimodal information in the car cockpit environment, it is possible to provide a rich data foundation for subsequent multi-task processing.

[0073] Step S110: Obtain a prompt instruction corresponding to each pre-configured monitoring task.

[0074] Specifically, according to the various functions required in the design of the smart cockpit, this application can pre-configure various types of monitoring tasks, including but not limited to Figure 2 In addition, in order to ensure the implementation of different monitoring tasks, the present application can also pre-configure a multi-task prompt instruction set, which includes a plurality of pre-set prompt instructions, and the prompt instructions correspond to the monitoring tasks. One or more prompt instructions can be set for one monitoring task, and the prompt instructions include query description information of the monitoring results or related information of the monitoring task.

[0075] Taking the speech recognition task as an example, the corresponding prompt command may include "Is there anyone talking in the car? What is the content of the conversation?"

[0076] In this embodiment, by configuring a multi-task prompt instruction set, necessary interactive guidance (ie, prompt instructions) can be provided for different monitoring tasks to guide the generation of monitoring results of the monitoring tasks in subsequent steps.

[0077] Specifically, for each configured monitoring task, a prompt instruction corresponding to each monitoring task may be obtained from a multi-task prompt instruction set.

[0078] Combination Figure 4 As shown, some optional prompt instructions in the multi-task prompt instruction set are illustrated, including but not limited to:

[0079] How many passengers are there in the car? What are their genders?

[0080] Did the lady sitting in the passenger seat (x1,y1,x2,y2) speak? What did she say?

[0081] Where is the driver (x3,y3,x4,y4) looking, is he distracted?

[0082] Introduce the breed, origin and main habits of the pet in the car?

[0083] Where exactly is the child in the back row? Is she asleep?

[0084] What color is the handbag of the lady in the passenger seat (x1,y1,x2,y2)?

[0085] Step S120: Generate a monitoring result for each monitoring task by using the multimodal information and the prompt instruction for each monitoring task.

[0086] Specifically, in this step, the multimodal information can be combined to generate the monitoring result of the monitoring task under the guidance of the prompt instruction of each monitoring task.

[0087] The monitoring results of the generated monitoring tasks may be in a unified format, for example, generating monitoring results in an understandable text format.

[0088] Combination Figure 4 As shown, for each prompt instruction in the above example, the corresponding generated monitoring results may include:

[0089] There were three passengers in the car, a man in the driver's seat, a woman in the front passenger seat, and a little girl in the back seat.

[0090] The lady sitting in the passenger seat (x1,y1,x2,y2) spoke, and what she said was "The weather is really nice today."

[0091] The driver (x3,y3,x4,y4) is looking straight ahead and is not distracted.

[0092] The pet in the car is a poodle, which originated in France. It is smart, elegant, friendly and easy to train, and has thick curly hair.

[0093] The position of the child in the back row is (x5, y5, x6, y6). She is not asleep but is playing with the puppy.

[0094] The handbag of the lady in the front passenger seat (x1,y1,x2,y2) is pink.

[0095] This step can be implemented through a pre-configured unified model framework. Exemplarily, a multi-task processing model (i.e., the above-mentioned unified model framework) can be pre-configured. On this basis, the acquired multimodal information and the prompt instructions of each monitoring task are sent to the multi-task processing model to obtain the monitoring results of each monitoring task output by the model.

[0096] Among them, the multi-task processing model is configured as follows: feature extraction and alignment are performed on each modal information in the input multi-modal information and the prompt instruction of each monitoring task to obtain the aligned features of each modal and the prompt instruction features; encoding and decoding operations are performed on the aligned features of each modal and the prompt instruction features to obtain the monitoring results of each monitoring task.

[0097] The multi-task processing model may adopt a variety of model structures, for example, a multi-modal large model may be adopted, etc. This embodiment introduces an optional structure of a multi-task processing model, which may include a feature alignment module, a feature encoding and decoding module.

[0098] Among them, the feature alignment module is used to extract and align the features of each input modal information and each monitoring task prompt instruction. Through feature alignment, the features of each modal and prompt instruction are effectively integrated to form sequence information, laying the foundation for subsequent processing steps. When performing feature alignment operations, the feature alignment module can use multiple methods such as feature splicing and feature fusion to achieve accurate alignment between different features.

[0099] Since the input includes multi-modal information and prompt instructions (generally text-based prompt instructions), the feature alignment module can include multiple sub-modules to perform feature extraction and alignment operations on information and prompt instructions of different modes, respectively. Figure 4 As shown, the feature alignment module may include sub-modules such as an audio processing module, a video processing module, an image processing module, a depth processing module, and a text processing module, which are used to process the information and prompt instructions of the corresponding modalities. For example, the audio modal information passes through the audio processing module to obtain audio coding features; the video modal information passes through the video processing module to obtain video coding features; the image modal information passes through the image processing module to obtain image coding features; the depth modal information passes through the depth processing module to obtain depth coding features; and the text modal information passes through the text processing module to obtain text coding features.

[0100] The feature encoding module is responsible for processing the aligned features of each modality and the prompt instruction features output by the feature alignment module. The feature encoding module converts and refines the features of each modality and the prompt instruction features, and maps them to a high-dimensional feature space in order to capture richer and deeper data representations. The decoding module decodes the features extracted by the encoding module to obtain text decoding features, which can then be converted into a more intuitive and operational form through a classification layer (such as a softmax layer), that is, the decoding obtains the monitoring results of each monitoring task. Combined with Figure 4 As shown, the monitoring results can be in text form. Through the processing of feature encoding and decoding modules, not only can the inherent characteristics of the input data be accurately reflected, but the decoding results also provide a reliable basis for the system to make subsequent decisions and responses.

[0101] The vehicle cabin monitoring method provided in the embodiment of the present application pre-arranges various types of monitoring tasks that need to be implemented in the vehicle cabin scene, such as voice recognition tasks, fatigue warning tasks, gesture control tasks, sight tracking tasks, emotion recognition tasks, etc. At the same time, a multi-task prompt instruction set is pre-configured, and the set contains multiple preset prompt instructions. The prompt instruction corresponds to the monitoring task, which may include inquiry description information of the monitoring result or related information of the monitoring task. Taking the voice recognition task as an example, the corresponding prompt instruction may include "Excuse me, is there anyone talking in the car at present, and what is he talking about?" On this basis, the multi-modal information monitored for the vehicle cabin environment is obtained. At the same time, for each type of monitoring task, the corresponding prompt instruction is obtained from the multi-task prompt instruction set, and the monitoring result of each monitoring task is predicted and generated using the simultaneous instruction and the obtained multi-modal information. Obviously, this application provides a unified framework that can integrate different monitoring tasks in the car cockpit. When generating the monitoring results of each monitoring task, the acquired multimodal information and the prompt instructions corresponding to the monitoring task can be used. Different monitoring tasks can share multimodal information, give full play to the complementary role of each modal information, optimize the complementarity between different monitoring tasks, and thus improve the overall monitoring performance. In addition, this application can realize cross-task processing through a unified framework, without the need to deploy each independent task function module separately, thereby reducing the burden of deploying related functional models.

[0102] In some possible implementations, after obtaining the monitoring result of each monitoring task, it is possible to further perform a response operation matching the monitoring result based on the monitoring result and in accordance with a configured control strategy.

[0103] Specifically, for different types of monitoring tasks, matching control strategies can be configured according to the different monitoring results. For example, for fatigue monitoring tasks, if the monitoring results indicate that the driver shows signs of fatigue, a warning signal can be triggered, such as outputting a warning signal through an audio module to remind the driver to stop and rest, or the car can be controlled to slow down or stop. For another example, for emotion perception monitoring tasks, if the monitoring results indicate that the driver's current emotion is happy, the sound effect parameters of the currently playing music can be actively adjusted, such as increasing the pitch level, speeding up the playing rhythm, etc., to create a more exciting and active auditory experience.

[0104] The above-mentioned process of executing the response operation matching the monitoring result can be realized by the driver and passenger monitoring and interactive brain, combined with Figure 4 As shown, the driver and passenger monitoring and interaction brain can adopt the vehicle system or other functional modules.

[0105] In this embodiment, by further adding post-processing operations on the monitoring results, that is, executing response operations matching the monitoring results, the performance of the smart cockpit can be further improved, and the safety, comfort and intelligence of driving can be improved.

[0106] In some possible implementations, the present application may further identify the interaction intention of the user in the car cabin based on the monitoring results of determining the monitoring results of the above-mentioned types of monitoring tasks, and then output a response to the user's interaction intention.

[0107] Among them, interaction intentions include but are not limited to voice or text inquiries, visual (gesture, line of sight) interactions, or multimodal interactions that combine various modal information.

[0108] In this embodiment, the user's interaction intention is identified with the help of the detection results of the monitoring task. In the specific implementation process, the monitoring task of the target type related to the identification of the user's interaction intention can be selected according to the type of monitoring task, and then the user's interaction intention can be identified based on the monitoring results of the target type monitoring task. For example, for voice interaction intention, it can be identified whether the user has voice interaction intention based on the monitoring results of the voice recognition monitoring task; for gesture interaction intention, it can be identified whether the user has gesture interaction intention based on the monitoring results of the gesture recognition monitoring task, etc.

[0109] After identifying the user's interaction intention, a response matching the user's interaction intention may be determined and then output.

[0110] Taking gesture interaction intention as an example, if it is understood based on the monitoring results that the user wants to increase the air-conditioning temperature, a first air-conditioning temperature adjustment instruction can be output to enable the air-conditioning temperature control module to increase the air-conditioning temperature.

[0111] Further optionally, before the above step of outputting a response to the user's interaction intention, the following step may be further added:

[0112] Identify the identity information of the user who currently has the intention to interact.

[0113] Exemplarily, the portrait information of the user who currently has the intention to interact can be obtained through an image sensor, and then the user's identity information can be identified based on a face recognition algorithm. Alternatively, the audio data of the user who currently has the intention to interact can be collected through a microphone, and then the user's identity information can be identified based on a voiceprint recognition algorithm, etc.

[0114] After identifying the identity information of the user who currently has the interaction intention, a response matching the interaction intention of the user and the identity information can be determined and output.

[0115] Taking the above-mentioned gesture interaction intention as an example, if it is understood based on the monitoring results that the user wants to increase the air-conditioning temperature, the identity of the current user can be identified, the air-conditioning temperature value preferred by the user with this identity can be obtained, and then the first air-conditioning temperature adjustment instruction can be output to enable the air-conditioning temperature control module to adjust the temperature to the temperature value preferred by the user.

[0116] By adding the identity recognition process for users with interaction intentions, personalized and accurate feedback on interaction intentions can be achieved, thereby improving the quality of interaction.

[0117] In some embodiments of the present application, the training process of the multi-task processing model in the aforementioned embodiments is further described.

[0118] In the training process of the multi-task processing model, the training data of the model is first obtained, including training samples and sample labels.

[0119] The training samples may include: multimodal training samples monitored from the car cabin scene and question description samples for the multimodal training samples. The sample labels include: answer text labels that match the question description samples.

[0120] The multimodal training samples may include various modal information including audio, image, video, depth signal, and physiological signal. The problem description samples of the multimodal training samples may be regarded as prompt instructions for proposing a problem description for the multimodal training samples.

[0121] For ease of understanding, this embodiment only takes image modality training samples as an example for explanation:

[0122] Combination Figure 5 , the left side is the image training sample, which can be represented as an image or video frame captured by the built-in camera in the car cockpit. The right side is the problem description sample and sample label designed for the image training sample. The problem description sample includes but is not limited to:

[0123] A first problem description sample is requested for a global description of the image training sample;

[0124] A second problem description sample is requested to perform a local description on the image training sample;

[0125] A third question describing a sample that asks questions about the visual content of the image training sample.

[0126] Correspondingly, the answer text label matching the first question description sample may be a global description text of the image training sample;

[0127] The answer text label matched with the second question description sample may be a local description text of the image training sample;

[0128] The answer text label that matches the third question description sample is the answer text corresponding to the question.

[0129] Combination Figure 5 The text part on the right illustrates the first question description sample and its answer text label, the second question description sample and its answer text label, and the third question description sample and its answer text label for the left image.

[0130] It should be noted that Figure 5 Taking image modality training samples as an example, the training samples in the actual training process can include training samples of multiple modalities, and the problem description samples designed for multimodal training samples can cover various types of monitoring tasks and active interaction tasks as much as possible, including but not limited to: spatial perception of images (including perception of the entire image and local areas), behavior prediction in video sequences, voiceprint recognition, natural language understanding and other key tasks.

[0131] For example, for the task of global monitoring of the distribution of passengers in the car, the sample description of the question may include "How many passengers are there in the car?"; for the task of detailed monitoring of specific areas, the sample description of the question may include "Where is the child in the back seat sitting, is he or she in a safety seat?"; for the speech recognition monitoring task, the sample description of the question may include "Is there anyone in the back seat who has spoken, and what did they say?"; for the task of monitoring driver fatigue or distraction, the sample description of the question may include "Is the driver focusing on the front, and are his eyes closed?". In the active interaction task, the sample description of the question may include "The passenger in the front seat asked a question about XX, how should I respond?", etc.

[0132] After the training data is obtained as described above, the multi-task processing model can be trained using the training data. Specifically, the multimodal training samples and the problem description samples can be input into the multi-task processing model, and the model uses an autoregressive paradigm to predict the output result.

[0133] Furthermore, the loss function value is calculated based on the output result and the answer text label, and the parameters of the multi-task processing model are updated according to the loss function value until the set model training end condition is reached.

[0134] The loss function of the model training process can include but is not limited to cross entropy loss, reinforcement learning loss, etc. Taking cross entropy loss as an example, the specific loss function is as follows:

[0135] Among them, y iIndicates the text character that needs to be predicted currently, y<i represents the text characters that have appeared in history, x represents the input multi-modal training sample, and P represents the likelihood probability of the threshold.

[0136] The inference process of the model can adopt a sequence decoding method, and the decoding methods include but are not limited to strategies such as greedy search and beam search.

[0137] By training the multi-task processing model with the above training data, the model learns the internal relationship between the combination of multi-modal information and the prompt instructions (problem description text) of each monitoring task and the monitoring results (i.e., the answer text labels corresponding to the problem description text), so as to facilitate the subsequent inference use of the multi-task processing model.

[0138] The automotive cockpit monitoring device provided in the embodiments of the present application will be described below. The automotive cockpit monitoring device described below can be mutually referred to with the automotive cockpit monitoring method described above.

[0139] See Figure 6 , Figure 6 which is a schematic structural diagram of an automotive cockpit monitoring device disclosed in the embodiments of the present application.

[0140] As Figure 6 shown, the device may include:

[0141] A multi-modal information acquisition unit 11, configured to acquire multi-modal information monitored for the automotive cockpit environment;

[0142] A prompt instruction acquisition unit 12, configured to acquire a prompt instruction corresponding to each of the monitoring tasks in a configured multi-task prompt instruction set according to various types of monitoring tasks configured in advance, where the prompt instruction corresponding to any monitoring task includes inquiry description information about the monitoring result or related information of the monitoring task;

[0143] A monitoring result generation unit 13, configured to generate a monitoring result for each of the monitoring tasks by using the multi-modal information and the prompt instruction for each of the monitoring tasks.

[0144] A possible implementation is that the device of the present application may further include:

[0145] A monitoring result response unit, configured to perform a response operation matching the monitoring result according to a configured control strategy based on the monitoring result of each of the monitoring tasks.

[0146] A possible implementation is that the device of the present application may further include:

[0147] An interaction intention recognition unit, configured to recognize the interaction intention of a user in the automotive cockpit based on the monitoring result of a preset target type monitoring task.

[0148] The interaction intention response unit is used to output a response to the interaction intention of the user.

[0149] In a possible implementation, the device of the present application may further include:

[0150] An identity recognition unit, used to recognize the identity information of the user who currently has the interaction intention before the above-mentioned interaction intention response unit outputs a response to the interaction intention of the user;

[0151] The process of the interaction intention response unit outputting a response to the user's interaction intention includes:

[0152] A response matching the interaction intention and identity information of the user is determined, and the response is output.

[0153] In a possible implementation, the process in which the monitoring result generating unit generates the monitoring result of each monitoring task by using the multimodal information and the prompt instruction of each monitoring task includes:

[0154] Sending the multimodal information and the prompt instruction of each monitoring task into the configured multi-task processing model to obtain the monitoring result of each monitoring task output by the model;

[0155] Among them, the multi-task processing model is configured as follows: feature extraction and alignment are performed on each modal information in the input multimodal information and the prompt instruction of each monitoring task to obtain the aligned modal features and prompt instruction features; encoding and decoding operations are performed on the aligned modal features and the prompt instruction features to obtain the monitoring results of each monitoring task.

[0156] In a possible implementation, the device of the present application may further include:

[0157] The model training unit is used to train the multi-task processing model, and the process includes:

[0158] Acquire multimodal training samples monitored for a car cabin scene, question description samples for the multimodal training samples, and matching answer text labels;

[0159] Inputting the multimodal training samples and the problem description samples into a multi-task processing model to obtain an output result of the model;

[0160] A loss function value is calculated based on the output result and the answer text label, and the parameters of the multi-task processing model are updated according to the loss function value until the set model training end condition is reached.

[0161] In a possible implementation, the multimodal training sample includes an image training sample, and the problem description sample includes at least one of the following:

[0162] A first problem description sample is requested to perform a global description on the image training sample;

[0163] A second problem description sample requesting a local description of the image training sample;

[0164] a third question description sample for asking questions about the visual content of the image training sample;

[0165] The answer text label matching the first question description sample is a global description text of the image training sample;

[0166] The answer text label matching the second question description sample is a local description text of the image training sample;

[0167] The answer text label that matches the third question description sample is the answer text corresponding to the question.

[0168] In a possible implementation, the multimodal information includes information in at least two of the following modes:

[0169] Information from audio, video, image, depth and physiological signal modalities.

[0170] The present application also provides an electronic device in an embodiment. Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to a fixed terminal such as a vehicle terminal, a vehicle controller, and the like. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0171] like Figure 7 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 to the random access memory (RAM) 603 to implement the vehicle cabin monitoring method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602 and the RAM 603 are connected to each other via a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0172] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0173] A vehicle is also provided in an embodiment of the present application, which includes the above-mentioned electronic device.

[0174] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the vehicle cabin monitoring methods provided in the embodiments of the present application.

[0175] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the vehicle cabin monitoring methods provided in the embodiments of the present application.

[0176] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0177] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0178] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0179] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0180] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A method for monitoring a vehicle cabin, characterized in that: include: Acquire multimodal information monitored from the vehicle cabin environment, wherein the multimodal information includes information in at least two of the following modalities: audio, video, image, depth signal, and physiological signal modality information; According to the pre-configured monitoring tasks of various types, a prompt instruction corresponding to each of the monitoring tasks is obtained in the configured multi-task prompt instruction set, wherein the prompt instruction corresponding to any monitoring task includes query description information of the monitoring result or related information of the monitoring task, and the monitoring tasks at least include: a spatial perception task of an image, a behavior prediction task in a video sequence, a speech recognition task, and a natural language understanding task; In combination with the complementary effects between the modal information in the multimodal information, the prompt instructions of each monitoring task are used to generate the monitoring results of each monitoring task, wherein in the process of generating the monitoring results of each monitoring task, different monitoring tasks share the multimodal information.

2. The method according to claim 1, characterized in that Also includes: Based on the monitoring result of each monitoring task, a response operation matching the monitoring result is executed according to the configured control strategy.

3. The method according to claim 1, characterized in that Also includes: Based on the monitoring results of the preset target type monitoring tasks, identify the interaction intention of the user in the car cabin; A response to the user's interaction intention is output.

4. The method according to claim 3, characterized in that Before outputting a response to the user's interaction intention, the method further includes: Identify the identity information of the user who currently has the intention to interact; The process of outputting a response to the user's interaction intention includes: A response matching the interaction intention and identity information of the user is determined, and the response is output.

5. The method according to claim 1, characterized in that The process of generating the monitoring result of each monitoring task by combining the complementary effects between each modal information in the multimodal information and using the prompt instruction of each monitoring task includes: Sending the multimodal information and the prompt instruction of each monitoring task into the configured multi-task processing model to obtain the monitoring result of each monitoring task output by the model; Among them, the multi-task processing model is configured as follows: feature extraction and alignment are performed on each modal information in the input multimodal information and the prompt instruction of each monitoring task to obtain the aligned modal features and prompt instruction features; encoding and decoding operations are performed on the aligned modal features and the prompt instruction features to obtain the monitoring results of each monitoring task.

6. The method according to claim 5, characterized in that The training process of the multi-task processing model includes: Acquire multimodal training samples monitored for a car cabin scene, question description samples for the multimodal training samples, and matching answer text labels; Inputting the multimodal training samples and the problem description samples into a multi-task processing model to obtain an output result of the model; A loss function value is calculated based on the output result and the answer text label, and the parameters of the multi-task processing model are updated according to the loss function value until the set model training end condition is reached.

7. The method according to claim 6, characterized in that The multimodal training samples include image training samples, and the problem description samples include at least one of the following: A first problem description sample is requested to perform a global description on the image training sample; A second problem description sample requesting a local description of the image training sample; a third question description sample for asking questions about the visual content of the image training sample; The answer text label matching the first question description sample is a global description text of the image training sample; The answer text label matching the second question description sample is a local description text of the image training sample; The answer text label that matches the third question description sample is the answer text corresponding to the question.

8. A vehicle cabin monitoring device, characterized in that: include: A multimodal information acquisition unit, used to acquire multimodal information monitored from the vehicle cabin environment, wherein the multimodal information includes information in at least two of the following modes: audio, video, image, depth signal, and physiological signal mode information; A prompt instruction acquisition unit is used to acquire prompt instructions corresponding to each of the pre-configured types of monitoring tasks in the configured multi-task prompt instruction set, wherein the prompt instruction corresponding to any monitoring task includes query description information of the monitoring result or related information of the monitoring task, and the monitoring tasks at least include: image spatial perception tasks, behavior prediction tasks in video sequences, speech recognition tasks, and natural language understanding tasks; A monitoring result generating unit is used to combine the complementary effects between the modal information in the multimodal information and utilize the prompt instructions of each monitoring task to generate the monitoring results of each monitoring task, wherein in the process of generating the monitoring results of each monitoring task, different monitoring tasks share the multimodal information.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the vehicle cabin monitoring method according to any one of claims 1 to 7.

10. An automobile, characterized in that: An electronic device comprising the electronic device described in claim 9.

11. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the vehicle cabin monitoring method according to any one of claims 1 to 7 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the vehicle cockpit monitoring method as described in any one of claims 1 to 7 is implemented.

Images