Cockpit teaching method, apparatus and vehicle

Abstract

A cockpit teaching method (200), device, and vehicle (100) applicable to the field of smart cockpits include: acquiring first data, including data collected by sensors of the vehicle (100) during driving, and / or images acquired by a dashcam of the vehicle (100) during driving (S210); acquiring first information associated with abnormal data based on the first data, the abnormal data indicating abnormal driving behavior and / or abnormal emotional state of the driver (S220); and, when the vehicle (100) is stationary, controlling a prompting device to prompt teaching content to the user based on the first information (S230). This method can be applied to smart cars or electric vehicles, providing teaching content to the user based on information related to abnormal data during driving when the vehicle (100) is stationary, helping the driver correct errors more quickly.

Description

Cockpit teaching methods, devices and vehicles Technical Field

[0001] This application relates to the field of smart cockpits, and more specifically, to a cockpit teaching method, apparatus, and vehicle. Background Technology

[0002] With the rapid development of intelligent vehicles, in-vehicle infotainment systems have not only made significant progress in driver assistance and entertainment but are also beginning to show great potential in the field of education. Young, novice drivers often have limited knowledge of traffic regulations and therefore desire in-vehicle systems that provide driving guidance and practice functions. However, existing in-vehicle instruction systems mostly rely on artificial intelligence (AI) to assist driving and provide audio prompts while in motion. Providing instruction while driving can easily distract users, thus affecting their driving safety. Summary of the Invention

[0003] This application provides a cockpit teaching method, device, and vehicle that can provide teaching content to users based on information related to abnormal data during driving when the vehicle is stationary. This helps drivers correct errors more quickly and also helps improve user driving safety.

[0004] In a first aspect, this application provides a cockpit teaching method, the method comprising: acquiring first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by a vehicle recorder during vehicle operation; acquiring first information associated with abnormal data based on the first data, the abnormal data indicating abnormal driving behavior of the driver and / or abnormal emotional state of the driver; and, when the vehicle is stationary, controlling a prompting device to prompt teaching content to the user based on the first information.

[0005] Based on the above technical solution, upon obtaining first information related to abnormal data during vehicle operation, the system can control a prompting device to provide instructional content to the user even when the vehicle is stationary. This avoids distracting the driver by providing prompts while driving, thus improving driving safety. Furthermore, obtaining instructional content from the first information related to abnormal data allows for more precise and faster correction of errors, improving driving skills and reducing the likelihood of subsequent traffic accidents.

[0006] In some possible implementations, the abnormal data can also indicate abnormal actions by occupants other than the driver (e.g., not wearing a seatbelt, extending a hand out of the window, etc.). Thus, even when the vehicle is stationary, the prompting device can be controlled to provide instructional content to the user based on information related to abnormal actions by other occupants.

[0007] In some possible implementations, the teaching content includes video clips associated with anomalous data, summaries and analyses of the reasons for the anomalous data, and question and answer (Q&A) related to the anomalous data.

[0008] For example, the teaching content includes Q&A related to the abnormal data, and the control prompting device prompts the user with the teaching content, including: the control prompting device (e.g., a sound-emitting device) utters a question to the user that is related to the abnormal data; and receives the user's response to the question.

[0009] In some possible implementations, the method further includes: evaluating the content of the response and controlling the prompting device to prompt the user with the evaluation result.

[0010] In some possible implementations, when the vehicle is stationary, the control prompting device prompts the user with teaching content based on the first information, including: when the vehicle switches from a driving state to a parked state, the control prompting device prompts the user with teaching content based on the first information.

[0011] In some possible implementations, when the vehicle switches from a driving state to a parked state, the control prompting device prompts the user with teaching content based on the first information, including: when the vehicle switches from a driving state to a parked state and the vehicle is in a parking space, the control prompting device prompts the user with teaching content based on the first information; or, when the vehicle switches from a driving state to a parked state and the vehicle is parked on the side of the road, the control prompting device prompts the user with teaching content based on the first information.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: controlling the display device to display a first simulated driving image based on the first information; and controlling the display device to display a second simulated driving image based on user operation commands for at least one of the components: steering wheel, accelerator pedal, gear shift, or brake pedal.

[0013] Based on the above technical solution, by having the user operate on at least one of the components such as the steering wheel, accelerator pedal, or brake pedal, the driver can intuitively feel the changes in the driving environment and the operational feedback during simulated driving practice, which helps to improve the driver's learning effect and mastery of driving skills.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, controlling the display device to display the first simulated driving image according to the first information includes: controlling the head-up display (HUD) to display the first simulated driving image according to the first information.

[0015] Based on the above technical solutions, HUD projection technology can enhance the visual effect, allowing the driver to sit in the driver's seat and look straight ahead, so that the driver can intuitively feel the changes in the driving environment and the operation feedback, which helps to improve the driver's learning effect and mastery of driving skills.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, before controlling the display device to display the second simulated driving image based on the user's operation on at least one of the components, such as the steering wheel, accelerator pedal, gear shift, or brake pedal, the method further includes: inputting the operation command and a preset simulated driving image into an image generation model to obtain the second simulated driving image.

[0017] Based on the above technical solution, by inputting operation commands and preset simulated driving images into the image generation model, an updated second simulated driving image can be obtained, which allows the driver to intuitively feel the changes in the driving environment and the operation feedback.

[0018] In some possible implementations, the preset simulated driving image includes a period of time before the anomalous data is generated and viewpoints and environmental elements during the period in which the anomalous data is generated.

[0019] In some possible implementations, the preset simulated driving image can be static, and the simulated driving image displayed on the display device does not need to be rendered and updated in real time based on user actions. In this way, since the preset simulated driving image is static, the vehicle or cloud server only needs to adjust the position of the virtual camera according to the user's actions, thus avoiding the need to re-render the entire environment for each operation. This approach can effectively reduce the computational overhead when calculating the simulated driving image, helping to improve the real-time performance and continuity of the simulated driving image displayed on the display device.

[0020] In some possible implementations, before controlling the display device to display the second simulated driving image, the method further includes: stitching the first simulated driving image and the second simulated driving image together.

[0021] In some possible implementations, the method also includes dynamically loading a newly generated simulated driving image when the user's actions change.

[0022] In some possible implementations, the method also includes dynamically predicting new simulated driving images when the user's actions change.

[0023] In some possible implementations, the method also includes the ability to delete the already played simulated driving image when a change in the user's action is detected.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the operation instruction and the preset simulated driving image are input into an image generation model to obtain the second simulated driving image, including: sending the operation instruction to a cloud server; and receiving the second simulated driving image sent by the cloud server based on the operation instruction and the preset simulated driving image.

[0025] Based on the above technical solution, the vehicle can send user operation commands when the vehicle is stationary to a cloud server. The cloud server can then generate an updated second simulated driving image based on the operation commands and the preset simulated driving image. The vehicle can receive the second simulated driving image sent by the cloud server and control the display device to display it. This leverages the high computing power of the cloud server to improve the real-time performance and continuity of the simulated driving image display.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, before controlling the display device to display the first simulated driving image based on the first information, the method further includes: obtaining a preset simulated driving image based on data from the first data during a period of time before the abnormal data was generated and during the period of time when the abnormal data was generated; and obtaining the first simulated driving image from the preset simulated driving image.

[0027] Based on the above technical solution, a preset simulated driving image can be obtained from the data in the first data, which includes the period before the abnormal data was generated and the data within the period during which the abnormal data was generated.

[0028] In some possible implementations, obtaining a preset simulated driving image based on data from the first data before the occurrence of the abnormal data and data from the period during which the abnormal data occurred includes: inputting the data from the first data before the occurrence of the abnormal data and data from the period during which the abnormal data occurred into an image generation model to obtain the preset simulated driving image.

[0029] In some possible implementations, the first simulated driving image may include one or more frames of the preset simulated image, starting from the initial frame.

[0030] In conjunction with the first aspect, in some implementations of the first aspect, controlling the display device to display a second simulated driving image based on a user's operation command to at least one of the components, such as the steering wheel, accelerator pedal, gear shift, or brake pedal, includes: controlling the display device to display the second simulated driving image and adjusting the feedback parameters of the driver's seat based on the operation command.

[0031] Based on the above technical solution, the feedback parameters of the driver's seat can be controlled according to the driver's operating commands. This provides the driver with more immersive and realistic driving feedback when the vehicle is stationary. This approach not only helps the driver better understand the causes of abnormal data but also strengthens memory through realistic tactile feedback, improving the effectiveness of simulated driving practice, enhancing driving skills, and thus reducing the likelihood of subsequent traffic accidents.

[0032] In conjunction with the first aspect, in some implementations of the first aspect, the feedback parameters of the seat in the driver's area are adjusted, including: adjusting one or more of the vibration frequency, vibration intensity, and tilt angle of the seat in the driver's area.

[0033] In conjunction with the first aspect, in some implementations of the first aspect, according to the first information, controlling the prompting device to prompt the user with teaching content includes: according to the first information, controlling the prompting device to prompt the user with a teaching course associated with the abnormal data; at the end of the teaching course, controlling the prompting device to prompt the user to perform a simulated driving exercise for the abnormal data; wherein controlling the display device to display a first simulated driving image includes: in response to receiving input from the user confirming the performance of the simulated driving exercise, controlling the display device to display the first simulated driving image.

[0034] Based on the above technical solution, the prompting device can first display a teaching course, allowing the driver to learn driving based on the course before prompting the driver to perform simulated driving practice targeting the abnormal data. This helps drivers to conduct targeted simulated driving practice after learning, improving the effectiveness of simulated driving practice, enhancing driving skills, and thus reducing the probability of subsequent traffic accidents.

[0035] In conjunction with the first aspect, in some implementations of the first aspect, obtaining first information associated with the abnormal data based on the first data includes: inputting the first data into a generation model to obtain one or more of the following: timestamp information of the abnormal data, teaching courses associated with the abnormal data, and information of targets associated with the abnormal data.

[0036] In some possible implementations, the method further includes: based on the timestamp information, obtaining data from the first data for a period of time before the occurrence of the abnormal data and data within the period of time during which the abnormal data occurred.

[0037] In some possible implementations, the course includes a summary and analysis of the causes of anomalous data, and / or, the aforementioned Q&A related to anomalous data.

[0038] In some possible implementations, the sensors include microphones and in-cabin sensors, which, based on the first data, acquire first information associated with the abnormal data, including: inputting the first data into a generative model to obtain information associated with the driver's abnormal emotional state.

[0039] In some possible implementations, the sensors include an accelerator pedal sensor and / or a brake pedal sensor, and based on the first data, first information associated with the abnormal data is obtained, including: inputting the first data into a generation model to obtain information associated with the driver's abnormal driving behavior.

[0040] In some possible implementations, the first data also includes map data, and the sensors include positioning sensors. Based on the first data, first information associated with abnormal data is obtained, including: inputting the first data into a generation model to obtain information associated with the driver's abnormal driving behavior.

[0041] In conjunction with the first aspect, in certain implementations of the first aspect, based on the first information, the control prompting device prompts the user with teaching content, including: performing one or more of the following: the control prompting device prompts that the abnormal data was not generated by the driver's misoperation or was generated by the driver's misoperation; the control display device displays a first image outside the cockpit during the time period in which the abnormal data occurred, or the control display device displays a second image outside the cockpit during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; while controlling the display device to display the first image or the second image, prompting the user with information about the target; controlling the display device to display the driver's emotional state when the abnormal data occurred, or controlling the display device to display the driver's emotional state during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; or controlling the display device to display the teaching course.

[0042] Secondly, this application provides a method for generating teaching content, the method comprising: acquiring first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by a dashcam during vehicle operation; inputting the first data into a generation model to obtain teaching content, the teaching content including one or more of the following: timestamp information of abnormal data, teaching courses associated with the abnormal data, and information of targets associated with the abnormal data, the abnormal data indicating abnormal driving behavior of the driver and / or abnormal emotional state of the driver.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the generative model includes a first sub-generative model and a second sub-generative model. The process of inputting the first data into the generative model to obtain teaching content includes: inputting the first data into the first sub-generative model to obtain a first generation result, which includes the timestamp information, a summary and / or analysis of the reasons for generating the abnormal data, and the target information; and inputting the first generation result into the second sub-generative model to obtain the teaching course.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes: when the vehicle is stationary, controlling the prompting device to provide prompts to the driver based on the teaching content.

[0045] In conjunction with the second aspect, in certain implementations of the second aspect, based on the teaching content, the control prompting device prompts the driver, including: performing one or more of the following: the control prompting device prompts that the abnormal data was not generated by the driver's misoperation or was generated by the driver's misoperation; the control display device displays a first image outside the cockpit during the time period in which the abnormal data occurred, or the control display device displays a second image outside the cockpit during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; while controlling the display device to display the first image or the second image, prompting the user with information about the target; controlling the display device to display the driver's emotional state when the abnormal data occurred, or controlling the display device to display the driver's emotional state during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; or controlling the display device to display the teaching course.

[0046] Thirdly, this application provides a method for generating simulated driving images, the method comprising: acquiring first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by a driving recorder during vehicle operation; inputting at least a portion of the first data into an image generation model to obtain a preset simulated driving image, the preset simulated driving image including a preset simulated driving image associated with abnormal data, the abnormal data indicating abnormal driving behavior of the driver and / or abnormal emotional state of the driver.

[0047] In conjunction with the third aspect, in some implementations of the third aspect, the preset simulated driving image includes a period of time before the abnormal data is generated and environmental information of the vehicle's surroundings during the period in which the abnormal data is generated.

[0048] In conjunction with the third aspect, in some implementations of the third aspect, the method further includes: controlling the display device of the terminal device to display the first simulated driving image in the preset simulated driving image.

[0049] In conjunction with the third aspect, in some implementations of the third aspect, the terminal device is a vehicle, and the display device of the control terminal device displays the first simulated driving image, including: controlling the HUD of the vehicle to display the first simulated driving image.

[0050] In conjunction with the third aspect, in some implementations of the third aspect, the method further includes: obtaining user operation instructions for the terminal device; inputting the operation instructions and the preset simulated driving image into the image generation model to obtain a second simulated driving image.

[0051] In conjunction with the third aspect, in some implementations of the third aspect, the method further includes: controlling the display device of the terminal equipment to switch from displaying the first simulated driving image to displaying the second simulated driving image.

[0052] In conjunction with the third aspect, in some implementations of the third aspect, the method further includes: inputting the operation instruction and the preset simulated driving image into the image generation model to obtain evaluation content or explanation content for the operation instruction; and controlling the prompting device of the terminal device to prompt the user with the evaluation content or explanation content.

[0053] In conjunction with the third aspect, in some implementations of the third aspect, inputting at least a portion of the data from the first data into a generation model to obtain a simulated driving image includes: inputting the data from the first data associated with the abnormal data into the image generation model to obtain the preset simulated driving image.

[0054] Fourthly, this application provides a cockpit teaching device, comprising: an acquisition unit for acquiring first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by a vehicle recorder during vehicle operation; the acquisition unit is further configured to acquire first information associated with abnormal data based on the first data, the abnormal data indicating abnormal driving behavior of the driver and / or abnormal emotional state of the driver; and a control unit for controlling a prompting device to prompt teaching content to the user based on the first information when the vehicle is stationary.

[0055] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the control unit is further configured to: control the display device to display a first simulated driving image based on the first information; and control the display device to display a second simulated driving image based on user operation commands to at least one of the steering wheel, accelerator pedal, or brake pedal.

[0056] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the control unit is specifically used to: control the head-up display (HUD) to display the first simulated driving image based on the first information.

[0057] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the device further includes: an image generation unit, used to input the operation command and a preset simulated driving image into an image generation model to obtain the second simulated driving image before the control unit controls the display device to display the second simulated driving image.

[0058] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the image generation unit is specifically used for: sending the operation instruction to the cloud server; and receiving the second simulated driving image sent by the cloud server based on the operation instruction and the preset simulated driving image.

[0059] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the acquisition unit is further configured to: acquire a preset simulated driving image based on data from the first data, including data from a period of time before the abnormal data was generated and data from the period of time during which the abnormal data was generated, before the control unit controls the display device to display the first simulated driving image; and acquire the first simulated driving image from the preset simulated driving image.

[0060] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the control unit is specifically used to: control the display device to display the second simulated driving image and adjust the feedback parameters of the driver's seat according to the operation instruction.

[0061] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the control unit is specifically used to: adjust one or more of the vibration frequency, vibration intensity, and tilt angle of the seat in the driver's area.

[0062] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the control unit is specifically configured to: control the prompting device to prompt the user with a teaching course associated with the abnormal data based on the first information; control the prompting device to prompt the user to perform a simulated driving exercise for the abnormal data at the end of the teaching course; and control the display device to display the first simulated driving image in response to receiving input from the user confirming the performance of the simulated driving exercise.

[0063] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the acquisition unit is specifically used to: input the first data into the generation model to obtain one or more of the following: the timestamp information of the abnormal data, the teaching course associated with the abnormal data, and the information of the target associated with the abnormal data.

[0064] In conjunction with the fourth aspect, in certain implementations of the fourth aspect, the control unit is specifically configured to perform one or more of the following: control the prompting device to indicate that the abnormal data was not generated by the driver's misoperation or was generated by the driver's misoperation; control the display device to display a first image outside the cabin during the time period in which the abnormal data occurred, or control the display device to display a second image outside the cabin during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; while controlling the display device to display the first image or the second image, prompt the user with information about the target; control the display device to display the driver's emotional state when the abnormal data occurred, or control the display device to display the driver's emotional state during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; or control the display device to display the instructional course.

[0065] Fifthly, this application provides an apparatus for generating teaching content, the apparatus comprising: an acquisition unit for acquiring first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by a vehicle recorder during vehicle operation; and a teaching content generation unit for inputting the first data into a generation model to obtain teaching content, the teaching content including one or more of the following: timestamp information of abnormal data, teaching courses associated with the abnormal data, and information of targets associated with the abnormal data, the abnormal data indicating abnormal driving behavior of the driver and / or abnormal emotional state of the driver.

[0066] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the generative model includes a first sub-generative model and a second sub-generative model. The teaching content generation unit is specifically used to: input the first data into the first sub-generative model to obtain a first generation result, which includes the timestamp information, a summary and / or analysis of the reasons for generating the abnormal data, and the target information; input the first generation result into the second sub-generative model to obtain the teaching course.

[0067] In conjunction with the fifth aspect, in some implementations of the fifth aspect, the device further includes: a control unit for controlling the vehicle's prompting device to provide prompts to the driver based on the teaching content when the vehicle is stationary.

[0068] In conjunction with the fifth aspect, in certain implementations of the fifth aspect, the control unit is specifically configured to perform one or more of the following: control the prompting device to indicate that the abnormal data was not generated by the driver's misoperation or was generated by the driver's misoperation; control the display device to display a first image outside the cabin during the time period in which the abnormal data occurred, or control the display device to display a second image outside the cabin during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; while controlling the display device to display the first image or the second image, prompt the user with information about the target; control the display device to display the driver's emotional state when the abnormal data occurred, or control the display device to display the driver's emotional state during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; or control the display device to display the instructional course.

[0069] Sixthly, this application provides an apparatus for generating simulated driving images, the apparatus comprising: an acquisition unit for acquiring first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by a vehicle recorder during vehicle operation; and an image generation unit for inputting at least a portion of the first data into an image generation model to obtain a preset simulated driving image, the preset simulated driving image including simulated driving images associated with abnormal data, the abnormal data indicating abnormal driving behavior of the driver and / or abnormal emotional state of the driver.

[0070] In conjunction with the sixth aspect, in some implementations of the sixth aspect, the preset simulated driving image includes a period of time before the abnormal data is generated and environmental information of the vehicle's surroundings during the period in which the abnormal data is generated.

[0071] In conjunction with the sixth aspect, in some implementations of the sixth aspect, the device further includes: a first control unit for controlling the display device of the terminal device to display the first simulated driving image in the preset simulated driving image.

[0072] In conjunction with the sixth aspect, in some implementations of the sixth aspect, the terminal device is a vehicle, and the first control unit is specifically used to: control the vehicle's HUD to display the first simulated driving image.

[0073] In conjunction with the sixth aspect, in some implementations of the sixth aspect, the acquisition unit is further configured to acquire user operation instructions for the terminal device; the image generation unit is further configured to input the operation instructions and the preset simulated driving image into the image generation model to obtain a second simulated driving image.

[0074] In conjunction with the sixth aspect, in some implementations of the sixth aspect, the device further includes a second control unit. The image generation unit is also used to input the operation command and the preset simulated driving image into the image generation model to obtain evaluation content or explanation content for the operation command. The second control unit is used to control the prompting device of the terminal device to prompt the user with the evaluation content or explanation content.

[0075] In conjunction with the sixth aspect, in some implementations of the sixth aspect, the image generation unit is specifically used to: input the data associated with the abnormal data in the first data into the image generation model to obtain the preset simulated driving image.

[0076] In a seventh aspect, this application provides a cockpit teaching device, the device including a processor and a memory, wherein the memory is used to store instructions, and the processor executes the instructions stored in the memory to cause the device to perform any of the possible methods in the first aspect.

[0077] Eighthly, this application provides an apparatus for generating instructional content, the apparatus including a processor and a memory, wherein the memory is used to store instructions, and the processor executes the instructions stored in the memory to cause the apparatus to perform any of the possible methods in the second aspect.

[0078] Ninthly, this application provides an apparatus for generating simulated driving images, the apparatus including a processor and a memory, wherein the memory is used to store instructions, and the processor executes the instructions stored in the memory to cause the apparatus to perform any of the possible methods in the third aspect.

[0079] In a tenth aspect, this application provides a cockpit teaching system, which includes a computing platform and a prompting device, wherein the computing platform includes any of the possible devices in the fourth or seventh aspect.

[0080] In the eleventh aspect, this application provides a vehicle that includes any one of the possible devices of the fourth to ninth aspects, or includes the system described in the tenth aspect.

[0081] In a twelfth aspect, this application provides a cloud server, which includes any one of the possible devices of the fifth, sixth, eighth, or ninth aspects.

[0082] In a thirteenth aspect, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform any one of the possible methods described in the first to third aspects above.

[0083] It should be noted that the above-mentioned computer program code can be stored in whole or in part on the first storage medium, wherein the first storage medium can be packaged together with the processor or packaged separately from the processor. This application embodiment does not specifically limit this.

[0084] In a fourteenth aspect, this application provides a computer-readable medium storing program code that, when run on a computer, causes the computer to perform any of the possible methods described in the first to third aspects above.

[0085] In a fifteenth aspect, this application provides a chip system including a processor for calling a computer program or computer instructions stored in a memory to cause the processor to perform any of the possible methods described in the first to third aspects above.

[0086] In conjunction with aspect fifteen, in one possible implementation, the processor is coupled to the memory via an interface.

[0087] In conjunction with aspect fifteen, in one possible implementation, the chip system further includes a memory in which computer programs or computer instructions are stored.

[0088] In a sixteenth aspect, this application provides a chip including circuitry for performing any of the possible methods described in the first to third aspects above. Attached Figure Description

[0089] Figure 1 is a functional block diagram of a vehicle provided in an embodiment of this application.

[0090] Figure 2 is a schematic flowchart of the cockpit teaching method provided in the embodiments of this application.

[0091] Figure 3 is a schematic diagram of the teaching courses obtained through the data collection and classification model and the course generation model in the embodiments of this application.

[0092] Figure 4 is a schematic flowchart of a method for generating teaching content provided in an embodiment of this application.

[0093] Figure 5 is a schematic block diagram of the method for generating simulated driving images provided in an embodiment of this application.

[0094] Figure 6 is a schematic flowchart of the abnormal data collection method provided in the embodiments of this application.

[0095] Figure 7 is a schematic diagram of preprocessing in multimodal scene understanding provided in an embodiment of this application.

[0096] Figure 8 is a schematic diagram of feature fusion in multimodal scene understanding provided in an embodiment of this application.

[0097] Figure 9 is a schematic diagram of processing the feature vector after fusing historical time and the feature vector after fusing current time according to an embodiment of this application.

[0098] Figure 10 is a schematic diagram of cockpit instruction when the vehicle is stationary, according to an embodiment of this application.

[0099] Figure 11 illustrates the teaching process when the vehicle is stationary, as provided in an embodiment of this application.

[0100] Figure 12 is a schematic diagram of the vehicle-mounted teaching system provided in an embodiment of this application.

[0101] Figure 13 is a graphical user interface (GUI) provided in an embodiment of this application.

[0102] Figure 14 is another GUI provided in an embodiment of this application.

[0103] Figure 15 is another GUI provided in an embodiment of this application.

[0104] Figure 16 is another GUI provided in an embodiment of this application.

[0105] Figure 17 is another GUI provided in an embodiment of this application.

[0106] Figure 18 is another GUI provided in an embodiment of this application.

[0107] Figure 19 is another GUI provided in an embodiment of this application.

[0108] Figure 20 is another GUI provided in an embodiment of this application.

[0109] Figure 21 is a schematic block diagram of the cockpit teaching device provided in an embodiment of this application.

[0110] Figure 22 is a schematic block diagram of the teaching content generation device provided in the embodiments of this application.

[0111] Figure 23 is a schematic block diagram of a device for generating simulated driving images provided in an embodiment of this application. Detailed Implementation

[0112] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. "At least one" refers to one or more. For example, "at least one of A and B," similar to "A and / or B," describes the association relationship between related objects, indicating that three relationships can exist. For example, at least one of A and B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0113] The prefixes such as "first" and "second" used in this application embodiment are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this application embodiment does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not constitute unnecessary restrictions due to the use of such prefixes. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0114] Figure 1 is a functional block diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 may include a sensing system 110, a computing platform 120, and a display device 130. The sensing system 110 may include one or more sensors for sensing information about the environment surrounding the vehicle 100. For example, the sensing system 110 may include a positioning system, which may be a Global Positioning System (GPS), a BeiDou system, or another positioning system. As another example, the sensing system 110 may include one or more of an inertial measurement unit (IMU), an accelerometer, a lidar, millimeter-wave radar, ultrasonic radar, and a camera device. For example, the accelerometer may include a sensor for detecting acceleration signals from the air suspension system, or it may include a sensor for detecting ESC acceleration signals.

[0115] Some or all of the functions of vehicle 100 can be controlled by computing platform 120. Computing platform 120 may include one or more processors, such as processors 121 to 12n (n being a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. In addition, the computing platform 120 may also include a memory for storing instructions. Some or all of the processors 121 to 12n can call the instructions in the memory to implement the corresponding functions.

[0116] The in-cabin display devices 130 are mainly divided into two categories: the first is the in-vehicle display screen; the second is the projection display screen, such as the HUD. The in-vehicle display screen is a physical display screen and an important component of the in-vehicle infotainment system. Multiple displays can be installed in the cabin, such as the digital instrument cluster display, the central control screen, the display screen in front of the front passenger (also known as the front-seat passenger), the display screen in front of the left rear passenger, the display screen in front of the right rear passenger, and even the car windows can be used as displays. The head-up display, also known as a head-up display system, is mainly used to display driving information such as speed and navigation on a display device in front of the driver (such as the windshield). This reduces the driver's eye-shift time, avoids pupil changes caused by eye-shifting, and improves driving safety and comfort. HUDs include, for example, combiner-HUD (C-HUD) systems, windshield-HUD (W-HUD) systems, and augmented reality HUD (AR-HUD) systems. It should be understood that HUDs can also evolve into other types of systems as technology progresses, and this application does not limit them.

[0117] The above description of the display device 130 uses an in-vehicle display screen and a projection display screen as examples, but the embodiments of this application are not limited thereto. For example, the display device 130 can also be a light display screen or a projection screen.

[0118] As mentioned earlier, with the rapid development of intelligent vehicles, in-vehicle infotainment systems have not only made significant progress in driver assistance and entertainment, but are also beginning to show great potential in the field of education. Young, novice drivers often have limited knowledge of traffic regulations and therefore desire in-vehicle systems that provide driving guidance and practice functions. However, existing in-vehicle instruction systems mostly rely on artificial intelligence (AI) to assist driving and provide audio prompts while in motion. Providing instruction while driving can easily distract users, thus affecting their driving safety.

[0119] This application provides a cockpit teaching method, device, and vehicle. When the vehicle is stationary, it can provide teaching content to the user based on abnormal data during driving, which helps the driver correct mistakes more quickly and improves the user's driving safety.

[0120] Figure 2 shows a schematic flowchart of the cockpit teaching method 200 provided in an embodiment of this application. The method 200 can be executed by the vehicle 100; or, the method 200 can be executed by the computing platform 120; or, the method 200 can be executed by a processor, circuit, or chip in the computing platform 200. The method 200 includes:

[0121] S210, acquire first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by the vehicle's dashcam during vehicle operation.

[0122] For example, the sensor may include an external sensor, such as an external camera, lidar, millimeter-wave radar, etc.

[0123] For example, the sensor may include a sensor located inside the cockpit, such as an in-cabin camera.

[0124] S220, based on the first data, obtain first information associated with the abnormal data, the abnormal data indicating abnormal driving behavior of the driver and / or abnormal emotional state of the driver.

[0125] Optionally, the sensors include cameras and radars (e.g., lidar, millimeter-wave radar, etc.) outside the cockpit. Based on the first data, first information associated with the abnormal data is obtained, including: inputting the first data into a generation model to obtain information associated with the driver's abnormal driving behavior.

[0126] Optionally, the sensors include microphones and sensors in the cockpit, and based on the first data, first information associated with the abnormal data is obtained, including: inputting the first data into a generation model to obtain information associated with the driver's abnormal emotional state.

[0127] Optionally, the sensors include an accelerator pedal sensor and / or a brake pedal sensor. Based on the first data, first information associated with the abnormal data is obtained, including: inputting the first data into a generation model to obtain information associated with the driver's abnormal driving behavior.

[0128] Optionally, the first data may also include map data, and the sensor may include a positioning sensor. Based on the first data, first information associated with abnormal data may be obtained, including: inputting the first data into a generation model to obtain information associated with the driver's abnormal driving behavior (e.g., wrong driving route).

[0129] Optionally, based on the first data, first information associated with the abnormal data is obtained, including: inputting the first data into the generation model to obtain one or more of the following: timestamp information of the abnormal data, teaching courses associated with the abnormal data, and information of targets associated with the abnormal data.

[0130] Optionally, the timestamp information of the abnormal data can be used to determine the start and end times of the abnormal data generation in the first data.

[0131] Optionally, the method 200 further includes: based on the timestamp information, obtaining data from the first data for a period of time before the abnormal data was generated and data within the period of time during which the abnormal data was generated.

[0132] Optionally, the teaching courses associated with the anomalous data may include a summary and analysis of the causes of the anomalous data, and / or, a Q&A session related to the anomalous data.

[0133] For example, the abnormal data could be caused by a driver crossing a solid line to change lanes. The reasons for this abnormal data could include: the driver's inaccurate judgment of lane markings when changing lanes along a solid line; the driver being fatigued when changing lanes along a solid line; or the driver being distracted when changing lanes along a solid line.

[0134] For example, the analysis generated by this abnormal data may include laws and regulations related to lane changes on solid lines, the risks of lane changes on solid lines, and how to avoid lane changes on solid lines.

[0135] Questions associated with abnormal data may include "How many points will be deducted for crossing a solid line?", "What are the risks of crossing a solid line?", and "How can I avoid crossing a solid line?". The vehicle can control an in-cabin audio device to emit voice messages corresponding to one or more of these questions. After receiving the voice message, the user can respond with their voice. Optionally, the vehicle can also evaluate the user's voice response.

[0136] Optionally, taking the first data as an example, which includes a first image captured by an external camera and / or a second image captured by a dashcam, information about the target associated with the abnormal data can be used to mark the relevant target in the first or second image for a period of time before the abnormal data was generated and during the period when the abnormal data was generated.

[0137] For example, the abnormal data could be a driver crossing a solid line to change lanes, and the target could be the solid line and the wheels close to it.

[0138] For example, the abnormal data could be that a driver has selected the wrong branch road at a highway intersection, and the target could be a road sign at the fork in the road.

[0139] For example, the anomalous data could be the driver's state of anxiety (e.g., the driver's heart rate increases), and the target could be a target in the vicinity (e.g., a large truck) that caused the state of anxiety.

[0140] Optionally, the generative model includes a first sub-generative model and a second sub-generative model. Inputting the first data into the generative model to obtain teaching content includes: inputting the first data into the first sub-generative model to obtain the first generation result, which includes timestamp information, a summary and / or analysis of the reasons for generating the abnormal data, and information about the target; inputting the first generation result into the second sub-generative model to obtain the teaching course.

[0141] For example, Figure 3 shows a schematic diagram of obtaining a teaching course through a data acquisition and classification model and a course generation model in an embodiment of this application.

[0142] The first sub-generation model described above can also be called a data acquisition and classification model, and the second sub-generation model can also be called a course generation model. By inputting the first data into this data acquisition and classification model, timestamp information (representing the time points when the data acquisition and classification model considers anomalous data valuable) and a summary and analysis of the reasons associated with each anomalous data can be obtained. Optionally, the data acquisition and classification model can also output the target associated with each anomalous data.

[0143] By inputting the data from the data collection and classification model into the course generation module, a teaching course on the correct operation corresponding to the abnormal data can be output, including Q&A related to the abnormal data. Optionally, the course generation module can also output video clips related to the abnormal data and reminders for simulated driving exercises based on the abnormal data.

[0144] The summary and analysis of the reasons for the above data collection and classification model output, as well as video clips related to abnormal data, can also be used as part of the teaching course.

[0145] Optionally, if the output of the data acquisition and classification model includes targets associated with each anomalous data point, the targets can be marked in the video clips related to the anomalous data output by the course generation module (e.g., displayed by a red box).

[0146] S230, when the vehicle is stationary, the control prompting device prompts the user with teaching content based on the first information.

[0147] In this embodiment, upon obtaining first information associated with abnormal data during vehicle operation, the prompting device can be controlled to provide instructional content to the user based on this first information when the vehicle is stationary. This avoids distracting the driver by providing prompts while driving. Furthermore, obtaining instructional content from the first information related to the abnormal data allows for more precise and faster correction of errors by the driver, improving driving skills and reducing the likelihood of subsequent traffic accidents, thus enhancing user safety.

[0148] Optionally, based on the first information, the control prompting device prompts the user with teaching content, including: performing one or more of the following (1)-(5):

[0149] (1) The control prompt device indicates that the abnormal data was not caused by the driver's misoperation or was caused by the driver's misoperation.

[0150] For example, consider an anomaly where a driver crosses a solid line to change lanes. The data acquisition and classification model can also output whether this lane-changing behavior was caused by driver error.

[0151] For example, if the data acquisition and classification model output indicates that the vehicle is traveling at high speed and the driver crosses a solid line to change lanes in order to avoid an obstacle in front of the vehicle (e.g., a pedestrian suddenly appearing in front of the vehicle), then it can be determined that the lane change across the solid line was not caused by the driver's misoperation. In this case, the prompting device can be left uncontrolled to provide the driver with instructional content.

[0152] For example, if the data collection and classification model outputs a result indicating that the vehicle was traveling at low speed and the driver crossed a solid line to change lanes in order to control the vehicle and overtake the vehicle in front, then it can be determined that the lane change was caused by the driver's misoperation.

[0153] For example, if the data collection and classification model outputs results indicating that the vehicle changed lanes along a solid line due to driver fatigue or distraction, then it can be determined that the lane change was caused by driver error.

[0154] (2) The control display device displays a first image outside the cabin during the time period in which the abnormal data was generated, or the control display device displays a second image outside the cabin during a period of time before the abnormal data was generated and during the time period in which the abnormal data was generated.

[0155] For example, the start time T1 and end time T2 of generating a certain abnormal data can be determined based on the timestamp information mentioned above. In this way, when the vehicle is stationary, the display device can be controlled to display a first image of the outside of the cabin within the time period [T1, T2]; or, a second image of the outside of the cabin within the time period [T0, T2] can be displayed, where time T0 is before the start time T1.

[0156] (3) When controlling the display device to display the first image or the second image, prompt the user with information about the target.

[0157] For example, the abnormal data could be a driver crossing a solid line to change lanes, and the target could be the solid line and the wheel close to it. Thus, when the vehicle is stationary, the solid line and the wheel close to it can be marked in the first or second image displayed on the control display device (e.g., by displaying the solid line and the wheel with a red box).

[0158] (4) Control the display device to display the driver's emotional state when the abnormal data is generated, or control the display device to display the driver's emotional state during a period of time before the abnormal data is generated and during the period of time when the abnormal data is generated.

[0159] For example, the abnormal data could be the driver's anxiety. The data acquisition and classification model output indicates that the cause of this anxiety is the driver overtaking a large truck in front. The start time T4 and end time T5 of this anxiety can be determined based on the timestamp information. Thus, when the vehicle is stationary, the display device can be controlled to show the driver's emotional state during the time period [T4, T5]; or, the change in the driver's emotional state during the time period [T3, T5] can be shown, for example, the transition from never showing anxiety to showing anxiety, with time T3 preceding the start time T5.

[0160] (5) Control the display device to display the teaching course.

[0161] Optionally, the course may include a summary and analysis of the causes associated with the anomalous data, and / or a Q&A session related to the anomalous data.

[0162] For example, a summary and analysis of the causes associated with abnormal data can be shown in Table 1.

[0163] Table 1

[0164] For example, the teaching content includes Q&A related to the abnormal data. The control prompting device prompts the user with the teaching content, including: the control prompting device issuing a question to the user that is related to the abnormal data; and receiving the user's answer to the question.

[0165] Questions associated with abnormal data may include "How many points will be deducted for crossing a solid line?", "What are the risks of crossing a solid line?", and "How to avoid crossing a solid line?". The vehicle can control an in-cabin audio device to emit voice messages corresponding to one or more of these questions. After receiving the voice message, the user can provide a voice response. Optionally, the method 200 further includes: evaluating the voice response and controlling a prompting device to display the evaluation result to the user.

[0166] Optionally, when the vehicle is stationary, the control prompting device prompts the user with teaching content based on the first information, including: when the vehicle switches from a driving state to a parked state, the control prompting device prompts the user with teaching content based on the first information.

[0167] Optionally, when the vehicle switches from a driving state to a parked state, the control prompting device prompts the user with teaching content based on the first information, including: when the vehicle switches from a driving state to a parked state and the vehicle is detected to be in a parking space, the control prompting device prompts the user with teaching content based on the first information.

[0168] Optionally, the method 200 further includes: controlling the display device to display a first simulated driving image based on the first information; and controlling the display device to display a second simulated driving image based on the user's operation command for at least one of the components, such as the steering wheel, accelerator pedal, gear shift, or brake pedal.

[0169] Optionally, before the control display device displays the first simulated driving image according to the first information, the method further includes: obtaining a preset simulated driving image based on data from the first data, including data from a period of time before the abnormal data was generated and data from the period of time during which the abnormal data was generated; and obtaining the first simulated driving image from the preset simulated driving image.

[0170] Optionally, obtaining a preset simulated driving image based on data from the first data before the occurrence of the abnormal data and data from the period during which the abnormal data occurred includes: inputting the data from the first data before the occurrence of the abnormal data and data from the period during which the abnormal data occurred into an image generation model to obtain the preset simulated driving image.

[0171] For example, the image generation model can be located in a vehicle.

[0172] For example, the image generation model can be located on a cloud server. The vehicle can send data from the first data, including the period before the abnormal data occurred and the period during which the abnormal data occurred, to the cloud server, so that the cloud server can input the data into the image generation model to obtain the preset simulated driving image.

[0173] Optionally, the preset simulated driving image includes a period of time before the abnormal data was generated and environmental information around the vehicle during the period when the abnormal data was generated. For example, the environmental information includes multiple perspectives from a virtual camera and one or more environmental elements.

[0174] It should be noted that if the information processed in this application involves users' personal information (such as facial images, fingerprint information, voiceprint information and other biometric information, location trajectory information and personal contact information, etc.), the processing of users' personal information will be based on legality, and users will be fully informed and authorized, in accordance with the relevant laws and regulations on personal information protection in the country or region.

[0175] For example, the display and updating of the simulated driving image in the simulated driving exercise may include the following three stages:

[0176] (1) Comprehensive large-scale scene pre-generation

[0177] The vehicle can obtain data from the first set of data, including the period before the occurrence of the abnormal data and the period during which the abnormal data occurred, based on the aforementioned timestamp information. The vehicle can then input this data into an image generation model to obtain a preset simulated driving image. This preset simulated driving image output by the image generation model can also be referred to as a large-scale pre-generated scene. The image generation model pre-generates a complete large scene, including multiple perspectives and environmental elements. The pre-generated large scene can be static and does not require real-time rendering updates based on user actions.

[0178] The above large-scale scenes can be the preset simulated driving images mentioned above.

[0179] Optionally, the first simulated driving image may include one or more frames of the preset simulated image starting from the starting frame.

[0180] (2) Virtual camera view control

[0181] When the vehicle receives a user's operation command on at least one of the components—steering wheel, accelerator pedal, gear shift, or brake pedal—the viewpoint of the virtual camera in the larger scene can be controlled to change. For example, when the user presses the accelerator pedal, the virtual camera's viewpoint can be controlled to accelerate forward. Similarly, when the user presses the brake pedal, the virtual camera's viewpoint can be controlled to decelerate. Furthermore, when the user reverses the vehicle, the virtual camera's viewpoint can be controlled to move backward. And when the user turns the steering wheel, the virtual camera's viewpoint can be controlled to move in the direction of the steering wheel's rotation. In this way, the static parts of the rendering environment remain unchanged; only the viewpoint of the virtual camera needs to be controlled to change.

[0182] When the image generation model receives user commands (such as acceleration, deceleration, and steering commands), it can generate a feedback video that meets the operation requirements. This feedback video can simulate corresponding perspective changes based on a pre-generated large scene and the user's actions.

[0183] Since the pre-generated large scene is static, the image generation model only needs to adjust the virtual camera's perspective based on the user's input, thus avoiding the need to re-render the entire environment for each operation. This method effectively reduces computation and improves the system's real-time performance.

[0184] The feedback video after adjusting the virtual camera's perspective can be the second simulated driving image mentioned above.

[0185] (3) Real-time response storage and dynamic splicing

[0186] During the feedback video generation process, the image generation model dynamically loads the scene, similar to real-time scene loading in a game. Each action (such as acceleration, lane change, braking, etc.) triggers the generation of a corresponding feedback video. New feedback videos are dynamically generated in real time and stitched together with previous ones. The image generation model adjusts the feedback videos based on real-time changes in user actions. If the user's action changes (e.g., making a U-turn or turning), the image generation model dynamically loads new feedback videos, dynamically predicts new feedback videos, and deletes old feedback videos that are no longer needed.

[0187] In this embodiment, a large scene is pre-generated through an image generation model. When the user's operation command of using the steering wheel, accelerator pedal, brake pedal, or gear shift is detected, the virtual camera's perspective can be switched in the large scene based on the operation command. Finally, an updated simulated driving image is generated according to the operation command, which can reduce computational requirements and ensure real-time interaction.

[0188] Optionally, according to the first information, controlling the display device to display the first simulated driving image includes: according to the first information, controlling the head-up display (HUD) to display the first simulated driving image.

[0189] Based on the above technical solutions, HUD projection technology can enhance the visual effect, allowing the driver to sit in the driver's seat and look straight ahead, so that the driver can intuitively feel the changes in the driving environment and the operation feedback, which helps to improve the driver's learning effect and mastery of driving skills.

[0190] Optionally, before controlling the display device to display the second simulated driving image based on the user's operation on at least one of the components, such as the steering wheel, accelerator pedal, gear shift, or brake pedal, the method further includes: inputting the operation command and a preset simulated driving image into an image generation model to obtain the second simulated driving image.

[0191] In this embodiment of the application, by inputting operation instructions and preset simulated driving images into the image generation model, an updated second simulated driving image can be obtained, which allows the driver to intuitively feel the changes in the driving environment and the operation feedback.

[0192] Optionally, before controlling the display device to display the second simulated driving image, the method 200 further includes: stitching the first simulated driving image and the second simulated driving image together.

[0193] Optionally, the method 200 further includes: dynamically loading a newly generated simulated driving image when the user's operation changes.

[0194] Optionally, the method 200 further includes dynamically predicting new simulated driving images when a change in the user's operation is detected. This helps to further improve the real-time performance of the system and helps ensure that the driver sees continuous simulated driving images.

[0195] Optionally, the method 200 further includes deleting the already played simulated driving image when a change in the user's operation is detected. This helps save memory overhead on the terminal device.

[0196] Optionally, inputting the operation instruction and the preset simulated driving image into an image generation model to obtain the second simulated driving image includes: sending the operation instruction to a cloud server; and receiving the second simulated driving image sent by the cloud server based on the operation instruction and the preset simulated driving image.

[0197] Optionally, based on the user's operation command to at least one of the components, such as the steering wheel, accelerator pedal, gear shift, or brake pedal, the display device is controlled to display a second simulated driving image, including adjusting the feedback parameters of the driver's seat according to the operation command.

[0198] For example, the vehicle can calculate the seat feedback parameters corresponding to the driver's commands (such as braking, acceleration, and steering commands) and the dynamic changes in the simulated driving image. These parameters include, for instance, the seat's vibration frequency, intensity, and tilt angle. The onboard control system transmits these parameters to the seat's control module in real time, simulating physical feedback under different driving conditions. For example, in an emergency braking scenario, the seat tilts forward and vibrates accordingly, simulating a realistic braking experience. Combining the visual content displayed on the HUD with the seat's physical feedback enhances the driver's immersion. This allows the driver to not only visually perceive the simulated driving scenario but also physically experience feedback consistent with the scenario (such as the centrifugal force simulation during a sharp turn). During simulated driving practice, the seat feedback parameters are dynamically adjusted based on the driver's actual actions and reactions to ensure the accuracy and realism of the feedback, thereby helping the driver master driving skills more effectively.

[0199] Optionally, based on the first information, the control prompting device prompts the user with teaching content, including: based on the first information, controlling the prompting device to prompt the user with a teaching course associated with the abnormal data; at the end of the teaching course, controlling the prompting device to prompt the user to perform a simulated driving exercise for the abnormal data; wherein, controlling the display device to display the first simulated driving image includes: in response to receiving input from the user confirming the performance of the simulated driving exercise, controlling the display device to display the first simulated driving image.

[0200] In this embodiment, the prompting device can first display a teaching course, allowing the driver to learn to drive based on the course before prompting the driver to perform simulated driving practice targeting the abnormal data. This helps the driver understand the causes and analysis of the abnormal data, enabling targeted simulated driving practice, improving the effectiveness of the practice, enhancing the driver's skills, reducing the likelihood of subsequent traffic accidents, and ultimately improving user safety.

[0201] Figure 4 shows a schematic flowchart of a method 400 for generating instructional content according to an embodiment of this application. The method 400 includes:

[0202] S410, acquire first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by the vehicle's dashcam during vehicle operation.

[0203] Taking the method 400 executed by the cloud server as an example, obtaining the first data includes: receiving the first data sent by the vehicle.

[0204] S420, the first data is input into the generation model to obtain teaching content, which includes one or more of the following: timestamp information of the abnormal data, teaching courses associated with the abnormal data, and information of targets associated with the abnormal data. The abnormal data indicates abnormal driving behavior of the driver and / or abnormal emotional state of the driver.

[0205] Optionally, the generative model includes a first sub-generative model and a second sub-generative model. The process of inputting the first data into the generative model to obtain teaching content includes: inputting the first data into the first sub-generative model to obtain a first generation result, which includes the timestamp information, a summary and / or analysis of the reasons for generating the abnormal data, and the information of the target; and inputting the first generation result into the second sub-generative model to obtain the teaching course.

[0206] Optionally, the method 400 further includes: when the vehicle is stationary, controlling the prompting device to provide prompts to the driver based on the teaching content.

[0207] Optionally, based on the teaching content, the control prompting device prompts the driver, including performing one or more of the following: the control prompting device prompts that the abnormal data was not generated by the driver's misoperation or was generated by the driver's misoperation; the control display device displays a first image outside the cockpit during the time period in which the abnormal data occurred, or the control display device displays a second image outside the cockpit during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; while controlling the display device to display the first image or the second image, prompting the user with information about the target; controlling the display device to display the driver's emotional state when the abnormal data occurred, or controlling the display device to display the driver's emotional state during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; or controlling the display device to display the teaching course.

[0208] The implementation process of S420 above can be referred to the description in method 200 and Figure 3 above, and will not be repeated here.

[0209] Figure 5 shows a schematic block diagram of a method 500 for generating simulated driving images according to an embodiment of this application. The method 500 includes:

[0210] S510, acquire first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by the vehicle's dashcam during vehicle operation.

[0211] Taking the method 500 executed by the cloud server as an example, obtaining the first data includes: receiving the first data sent by the vehicle.

[0212] S520, at least a portion of the data in the first data is input into an image generation model to obtain a preset simulated driving image. The preset simulated driving image includes a preset simulated driving image associated with abnormal data, which indicates abnormal driving behavior and / or abnormal emotional state of the driver.

[0213] Optionally, the preset simulated driving image includes a period of time prior to the generation of the anomalous data and environmental information surrounding the vehicle during the period in which the anomalous data was generated. For example, the environmental information includes multiple viewpoints within the scene associated with the anomalous data and one or more environmental elements. These environmental elements can be determined from data in the first data that is associated with the anomalous data.

[0214] Optionally, the method 500 further includes: the display device of the control terminal device displays the first simulated driving image in the preset simulated driving image.

[0215] The aforementioned terminal devices can be vehicles, mobile phones, tablets, computers, augmented reality (AR) devices, or virtual reality (VR) devices, etc.

[0216] Taking the method 500 executed by a cloud server as an example, the display device of the control terminal device displays the first simulated driving image in the preset simulated driving image, including: sending the first simulated driving image to the terminal device.

[0217] Optionally, the terminal device is a vehicle, and the display device of the control terminal device displays the simulated driving image, including: controlling the HUD of the vehicle to display the first simulated driving image.

[0218] Optionally, the method 500 further includes: obtaining user operation instructions for the terminal device; inputting the operation instructions and the preset simulated driving image into the image generation model to obtain a second simulated driving image.

[0219] Taking the method 500 being executed by a cloud server as an example, obtaining the user's operation instructions for the terminal device includes: receiving the operation instructions sent by the terminal device; wherein, the method 500 further includes: sending the second simulated driving image to the terminal device.

[0220] Optionally, the method 500 further includes: inputting the operation instruction and the preset simulated driving image into the image generation model to obtain evaluation content or explanation content for the operation instruction; and controlling the prompting device of the terminal device to prompt the user with the evaluation content or explanation content.

[0221] For example, the evaluation content can indicate whether the user has passed the simulated driving exercise. If the user passes the simulated driving exercise, the vehicle can control the sound device to emit the voice message "This exercise is passed, congratulations," or "This exercise is passed, your ranking is in the top 10% of all users participating in the simulated driving exercise"; or, if the user fails the simulated driving exercise, the vehicle can control the sound device to emit the voice message "This exercise is failed, you can take it again."

[0222] Optionally, inputting at least a portion of the first data into a generation model to obtain a simulated driving image includes: inputting data from the first data associated with the abnormal data into the image generation model to obtain the preset simulated driving image.

[0223] The implementation process of S520 above can be referred to the description of method 200 above, and will not be repeated here.

[0224] Figure 6 shows a schematic flowchart of an anomaly data collection method 600 provided in an embodiment of this application. The method 600 includes:

[0225] S610 inputs the first data into the multimodal large model.

[0226] The above multimodal large models can include the generative models mentioned above.

[0227] S620 acquires information from the multimodal large model output that is associated with the driver's abnormal driving behavior.

[0228] For example, information associated with a driver’s abnormal driving behavior includes timestamps associated with the abnormal driving behavior, causal analysis and summaries associated with the abnormal driving behavior, objectives associated with the abnormal driving behavior, and teaching courses associated with the abnormal driving behavior.

[0229] Taking the method 600 executed by a vehicle as an example.

[0230] The vehicle can acquire data collected by sensors and / or images collected by a dashcam.

[0231] The vehicle can utilize multi-dimensional information both inside and outside the cabin. After preliminary processing and feature extraction by the preprocessing module, this information is input into a multimodal large model for deep fusion and understanding.

[0232] For example, consider the acquisition of images and depth information outside the vehicle's cabin.

[0233] Data source: External cameras capture road images, including road signs, traffic lights, and road markings. Meanwhile, radar sensors provide depth information, enhancing the spatial understanding capabilities of the images.

[0234] Feature Extraction: Vehicles can divide image data into fixed-size image patches and flatten them into vectors, which are then input into the vision transformer (ViT) model. ViT analyzes the relationships between these image patches through a self-attention mechanism, thereby extracting global features, including the shape, color, and text of road signs, as well as detailed information about traffic signs. ViT excels at handling long-range dependencies in images and can accurately capture and preserve global information in complex scenes.

[0235] Multimodal large model understanding and recognition: The vehicle fuses the RGB image features extracted by ViT with the depth information provided by radar sensors to generate an RGB-D data representation containing four channels. This RGB-D data representation not only includes the color and shape information of the image, but also incorporates depth information to provide richer spatial perception capabilities.

[0236] The fused RGB-D information is then passed to the multimodal large model. Through an alignment mechanism, the multimodal large model aligns the visual information (RGB-D) with textual information (such as traffic rules and road sign descriptions) to understand and recognize the content in the image. During the pre-training and fine-tuning phases, the multimodal large model has already acquired textual descriptions related to traffic regulations. Therefore, when recognizing a road sign or traffic sign, the multimodal large model compares and aligns it with the built-in textual descriptions to accurately understand the meaning of the road sign or sign.

[0237] Output and Prompts: Once recognition is complete, the vehicle compares the results with its built-in traffic regulation database to generate corresponding driving prompts or warnings. These prompts are delivered to the driver in real time through in-vehicle cabin prompt devices (e.g., in-vehicle audio or display devices) to ensure that the driver complies with traffic rules and drives safely.

[0238] S630 acquires information from the multimodal large model output that is associated with the driver's abnormal emotional state.

[0239] For example, let's take the acquisition of images inside the cockpit of a vehicle as an example.

[0240] Vehicles can capture the driver's physiological information (such as heart rate, respiration, and body temperature) as an independent modality and align it with text information (e.g., the text information corresponding to the user's voice command "Can I take the second lane from the left?") to capture the driver's emotional state during driving. The vehicle processes these physiological signals using multimodal large model technology to identify abnormal driving states and optimize the retrieval and processing of abnormal driving scenarios.

[0241] Data source: Real-time collection of the driver's physiological data through in-vehicle cameras, heart rate monitors on the steering wheel, and body temperature sensors.

[0242] Feature concatenation and embedding space projection: Vehicles can concatenate various modal features such as heart rate, respiration, and body temperature to generate a high-dimensional multimodal feature vector representing the driver's comprehensive physiological state at a specific moment. The concatenated feature vector is projected onto a shared embedding space so that features from different modalities can be aligned and fused. This process ensures that features from different modalities can be effectively compared within a unified semantic space, thereby establishing correlations between multimodal data.

[0243] Emotional Information Extraction and Model Training: The vehicle independently trains its model on each modal feature set, using a Long Short-Term Memory (LSTM) network to process time-series data such as heart rate, respiration, and body temperature. LSTM identifies abnormal emotional fluctuations or physiological states by capturing the contextual information of each modal feature changing over time.

[0244] After independent training, the vehicle can concatenate the prediction results of each modality to generate a comprehensive feature vector containing prediction information from each modality, representing the driver's overall physiological state. This comprehensive feature vector is then input into a separate LSTM model to capture information interactions and dependencies between modalities. The memory mechanism of LSTM enables it to effectively capture cross-modal temporal dependencies, thereby improving the accuracy of emotional state prediction.

[0245] For example, let's take the data collected by the vehicle's pedal sensors (accelerator pedal sensor, brake pedal sensor) as an example.

[0246] The pedal engagement frequency is a one-dimensional time-series signal, and the vehicle can use fully connected neural networks (FCNN) for feature extraction. FCNN excels at processing continuous signals without adjacent dependencies. By analyzing the frequency and force of pedal operation, it can identify abnormal driving behaviors, such as excessive or accidental pedal presses.

[0247] For example, consider the vehicle acquiring map data and data collected by positioning sensors.

[0248] The vehicle obtains its driving trajectory data through map data and location information displayed on the central control screen. This trajectory data is processed by an LSTM network, referencing historical driving trajectory data to detect trajectory deviations and abnormal route selections. For example, the vehicle can detect when it selects the wrong branch road at a highway exit and, combined with the timing of physiological signals (such as heart rate fluctuations), determine whether the driver is overly nervous or failed to anticipate correct road signs.

[0249] For example, Figure 7 illustrates a schematic diagram of preprocessing in multimodal scene understanding provided in an embodiment of this application. The generated model above may include a preprocessing module. The input to the preprocessing module may be user information, vehicle information, and environmental information, and the output may be driver state features, command audio features, navigation image features, vehicle state features, driving trajectory features, and visual features inside and outside the cockpit.

[0250] For example, Figure 8 illustrates a schematic diagram of feature fusion in multimodal scene understanding provided in an embodiment of this application. The above generative model may include a feature fusion module. The input to the feature fusion module may be driver state features, command audio features, navigation image features, vehicle state features, driving trajectory features, and visual features inside and outside the cockpit; the output may be a fused feature vector.

[0251] For example, Figure 9 shows a schematic diagram of processing the feature vector fused from historical moments and the feature vector fused from the current moment, as provided in an embodiment of this application. By inputting the feature vector fused from historical moments and the feature vector fused from the current moment into a multimodal large model, the multimodal large model can output descriptions of traffic trajectories and targets outside the cockpit related to abnormal data, as well as label and record abnormal data (abnormal scenarios).

[0252] For example, a vehicle can store detected fragments of abnormal data (including multimodal information such as driving operations, external environment, and driver status) as course materials.

[0253] For example, segments associated with anomalous data may include video clips associated with the anomalous data. To optimize the error retrieval process, the vehicle can further filter and prioritize stored segments. The vehicle can identify segments with greater educational value based on parameters such as the driver's physiological signals (e.g., heart rate, respiration) and operating frequency. For instance, the vehicle can prioritize recording anomalous data generated when the driver's heart rate is abnormally high; this anomalous data may reflect typical errors made by the driver under high pressure and has higher educational value. In the course material library, the vehicle prioritizes segments associated with anomalous data based on the severity and frequency of occurrence, as well as the driver's physiological reactions (or emotional changes), ensuring that key issues are highlighted and corrected in subsequent simulated teaching. Simultaneously, low-priority error segments can be cleaned up to optimize storage space and improve the overall teaching experience.

[0254] S640 determines whether the vehicle is in motion.

[0255] If the vehicle is in motion, execute S610; otherwise, execute S650.

[0256] S650, based on information associated with abnormal data, controls the prompting device to provide instructional content to the user.

[0257] For example, the teaching content may include video clips associated with anomalous data, summaries and analyses of the reasons associated with anomalous data, teaching courses associated with anomalous data, and objectives associated with anomalous data.

[0258] The above method 600 can generate a model in real time by inputting data collected by sensors and / or dashcams during vehicle operation, thereby obtaining information and teaching content related to the abnormal data. Alternatively, it can transmit the data collected by sensors and / or dashcams to a cloud server in real time. Since the cloud server inputs the data collected by sensors and / or dashcams into the model in real time, it obtains information and teaching content related to the abnormal data. The cloud server can then send this information and teaching content to the terminal device.

[0259] Figure 10 illustrates a schematic diagram of cockpit instruction when the vehicle is stationary, as provided in an embodiment of this application. The process includes:

[0260] S1001, The vehicle is detected to be stationary.

[0261] Optionally, detecting that the vehicle is stationary includes detecting that the vehicle is in a parked state.

[0262] Optionally, detecting that the vehicle is in a parked state includes: detecting that the vehicle is in a parking space and is in a parked state; or detecting that the vehicle is parked on the side of the road and is in a parked state.

[0263] S1002, when abnormal data exists while the vehicle is in motion, the control prompt device prompts the user with teaching content.

[0264] The above S1002 can be used to generate the teaching content by inputting the data collected by the sensors and / or driving recorder when the vehicle is in motion into the model when the vehicle is stationary.

[0265] Alternatively, the teaching content can be obtained by periodically inputting data collected by sensors and / or dashcams into the generative model while the vehicle is in motion.

[0266] For example, if abnormal data exists when the vehicle is in motion, the display device can be controlled to display video clips associated with the abnormal data when the vehicle is stationary, and the cause analysis and summary associated with the abnormal data can be broadcast through the sound device.

[0267] S1003, when the teaching content is completed, the control prompt device prompts the driver to conduct simulated driving practice.

[0268] S1004, based on data collected by sensors and / or a driving recorder while the vehicle is in motion, acquire a preset simulated driving image.

[0269] The process of obtaining the preset simulated driving image described above can be referred to the description in the above embodiments, and will not be repeated here.

[0270] S1005, the control display device displays the first simulated driving image.

[0271] For example, the first simulated driving image includes one or more frames of images that start from the initial frame in a preset driving image.

[0272] For example, the display device can be a HUD.

[0273] S1006, obtains user operation commands for the steering wheel, brake pedal, accelerator pedal or gear position.

[0274] S1007, according to the operation instruction, acquire the second simulated driving image.

[0275] The process of obtaining the second simulated driving image according to the operation instructions can be referred to the description in the above embodiments, and will not be repeated here.

[0276] S1008, the control display device displays the second simulated driving image.

[0277] Optionally, the method 1000 further includes: adjusting the feedback parameters of the seat to simulate the driving state according to the operation instruction.

[0278] Optionally, the method 1000 further includes: obtaining evaluation content or explanation content of the operation instruction based on the operation instruction.

[0279] The evaluation or explanation of the operation instruction obtained above can be found in the description in the above embodiments, and will not be repeated here.

[0280] Optionally, the method 1000 further includes: determining whether the user has passed the simulated driving practice (or, determining whether the simulated driving practice is qualified) based on the user's operation commands during the simulated driving practice. If the user passes the simulated driving practice, the teaching can be terminated; otherwise, the user can be prompted whether to repeat the teaching.

[0281] For example, Figure 11 illustrates the teaching process when the vehicle is stationary, as provided in an embodiment of this application.

[0282] The multimodal large model in this embodiment generates personalized driving instruction courses based on user input commands, vehicle parameters, and saved error fragments. For example, this can be achieved through the following steps:

[0283] Input Data Collection and Processing: In this embodiment, various real-time data can be dynamically collected during vehicle operation, including but not limited to speed, steering angle, braking force, accelerator / electric pedal position, and route selection. This data is used to analyze the driver's operating habits and behaviors in detail and identify potential abnormal data. The system's built-in knowledge base is associated with the detected abnormal data. For example, when a user makes a lane-changing error, the system can automatically retrieve and match teaching content related to correct lane-changing techniques. Based on the retrieved knowledge and the reasoning capabilities of the multimodal large model, the system then generates a customized teaching course. Furthermore, users can select specific learning objectives (such as lane-changing techniques, braking control, etc.) via voice commands or the central control screen to further customize the teaching content.

[0284] Multimodal Information Fusion and Alignment: In this embodiment, multimodal data (including driving operation behavior, in-vehicle and out-of-vehicle environment, driver physiological state, etc.) extracted from user commands, vehicle parameters, and recorded error fragments can be fused. This multimodal data is first mapped into a shared embedding space to ensure semantic and feature-level alignment and complementarity of each modality. The multimodal large model receives this fused data and, through an alignment mechanism, accurately matches driving operations with relevant teaching content in the knowledge base. This process ensures that the generated courses can provide efficient and personalized guidance tailored to the driver's specific needs and erroneous behaviors.

[0285] Scene Simulation and Video Generation: In this embodiment, a four-channel input can be constructed using RGB images, depth information, and vehicle operation data (such as steering wheel angle and speed) collected during driving, encompassing rich visual and spatial information. This multimodal data is input into a multimodal large model, which uses a self-attention mechanism and deep learning algorithms for data fusion and processing to generate a high-precision 3D driving scene. The system effectively enhances the realism, texture details, lighting effects, and expressiveness of rapid motion by combining depth information. The final generated 3D scene video not only includes visual effects but also simulates key factors in the dynamic environment (such as road conditions and traffic signs) to provide a realistic learning experience.

[0286] HUD Projection: The generated 3D simulated video is transmitted to the HUD via the in-vehicle system. The HUD can project the video content directly into the driver's field of vision using augmented reality (AR), allowing the driver to intuitively experience the simulated scenario in a real driving environment. During the simulated video playback, the driver can interact with the system via voice commands or the central control interface, such as switching course content, restarting the simulation, or adjusting learning parameters to improve the flexibility and relevance of learning.

[0287] Seat Parameter Processing and Feedback: In this embodiment, the system calculates seat feedback parameters corresponding to the driver's simulated operations (such as braking, acceleration, and steering) and the dynamic changes in the scene video. These parameters include the seat's vibration frequency, vibration intensity, tilt angle, etc., and are transmitted in real time to the seat feedback module through the vehicle control system to simulate physical feedback under different driving conditions. For example, in an emergency braking scenario, the seat will tilt forward and generate corresponding vibrations to simulate a real braking experience. By combining the visual content displayed on the HUD with the physical feedback of the seat, the system significantly enhances the driver's immersion. The driver can not only visually perceive the simulated scene but also feel physical feedback consistent with the scene through their body (such as the simulation of centrifugal force during a sharp turn). The system also has an intelligent adjustment function, dynamically adjusting the seat feedback parameters according to the driver's actual operation and reaction to ensure the accuracy and realism of the feedback, thereby helping the driver to master driving skills more effectively.

[0288] For example, Figure 12 shows a schematic diagram of an in-vehicle teaching system provided in an embodiment of this application.

[0289] With the rapid development of intelligent electric vehicle technology, the advancements of in-vehicle systems in areas such as driver assistance, entertainment, and education are particularly significant. In the field of driver training, this application provides a highly innovative in-vehicle teaching system that utilizes multimodal large-scale modeling technology to elevate traditional driver instruction to unprecedented levels. By acquiring real-time image, audio, and sensor data from inside and outside the vehicle, it dynamically interacts with the driver, records and analyzes erroneous driving behaviors, and then pushes personalized teaching content. This dynamic interaction not only helps improve the interactivity and engagement of driver learning but also enhances learning efficiency and flexibility, enabling drivers to gain a comprehensive learning experience in both actual and simulated driving. The in-vehicle teaching system in this application is not limited to providing simple error reminders but also has the ability to provide a seamless educational experience for drivers in both moving and stationary states. During driving, the in-vehicle teaching system can intelligently recognize road signs and traffic signs, explain relevant knowledge in real time, and issue warnings after the driver makes a mistake. The in-vehicle teaching system automatically records these erroneous behaviors and pushes targeted corrective suggestions and traffic regulation information at appropriate times. When the vehicle is stationary, the onboard teaching system will arrange teaching content and simulated driving exercises according to the driver's learning needs. The simulated driving exercises combine real driving data and simulated scenarios to provide comprehensive and detailed guidance and feedback, helping drivers quickly master complex driving skills.

[0290] The in-vehicle teaching system proposed in this application is based on multimodal large model technology, integrating advanced technologies from multiple fields such as language understanding, visual perception, and sensor data processing. Through joint learning and contrastive learning, the in-vehicle teaching system can deeply integrate various modal information with language, enabling the large model to possess advanced capabilities in understanding, analysis, and feedback. The in-vehicle teaching system can accurately identify driver commands, perceive changes in the driving environment, and generate corresponding teaching feedback in real time. Utilizing the deep learning capabilities of the multimodal large model, the in-vehicle teaching system can compare driver errors with correct operations during driving, generating simulated visual and audio feedback, allowing the driver to experience the consequences of incorrect operations in an immersive way, thereby gaining a deeper understanding and correcting driving behavior. The architecture of this in-vehicle teaching system can be shown in Figure 12.

[0291] System inputs include user commands, driver preferences, and raw data collected by sensors inside and outside the cockpit. This input data is pre-processed before being transmitted to the next module.

[0292] Multimodal scene understanding algorithms are responsible for processing information from various input sources and fusing and extracting multimodal information.

[0293] The simulation-based generative algorithm leverages the powerful real-world simulation capabilities of generative models to process data collected during vehicle operation, generating video and audio feedback including both correct and incorrect driving maneuvers. It not only recreates actual driving scenarios but also simulates the results of maneuvers in various driving situations, helping drivers deepen their understanding of driving behavior. The final output includes one or more of the following: a text-based description of the vehicle's interior and exterior environment and a real-time display of the driver's status; audio-based driving instruction and traffic regulations; video simulations of driving maneuvers displayed on a terminal device (e.g., the vehicle's HUD); and realistic feedback after simulated actions in the driver's seat, such as acceleration, braking, and the state after a collision.

[0294] The in-cabin equipment is responsible for outputting the processed information through in-vehicle devices, including HUD displays, seat movement feedback, and voice announcements. The system intelligently adjusts the output mode of the in-cabin equipment based on the driver's actual state, thereby simulating feedback in a real driving environment and providing a more immersive learning experience.

[0295] Multimodal information fusion: By fusing language understanding and multimodal data such as RGB-D images, the system can accurately understand the driving environment and the driver's needs, thereby providing more intelligent and accurate feedback.

[0296] Driver assistance: By acquiring real-time environmental data inside and outside the vehicle, it comprehensively analyzes the driver's breathing, body temperature, heart rate, pedal intervention frequency, and driving route to promptly detect abnormal situations, record them, and provide guidance. During interaction with the driver, it provides real-time road condition information and various traffic sign information to help the driver better understand complex traffic regulations.

[0297] Driver training: The system uses algorithms to detect driving status while driving and record violations or incorrect driving methods. When the vehicle is stationary, the data is processed and corresponding courses are generated. Using video, audio and image information generated by the generative model, the driver can return to the driving scenario, deeply understand the consequences of incorrect operation, and become safer through teaching.

[0298] Figures 13-17 show a set of graphical user interfaces (GUIs) provided in the embodiments of this application.

[0299] As shown in Figure 13, the central control screen can display navigation information from the map application while the vehicle is in motion, such as "keep to the right after 20 meters".

[0300] As shown in Figure 14, when the vehicle is determined to turn left into city A after passing the highway exit based on the vehicle's positioning sensor and map data, the central control screen can display the prompt message "The navigation route has been replanned for you".

[0301] The vehicle can input data collected by sensors and / or dashcams during driving into the generative model to obtain information associated with abnormal data (such as selecting the wrong branch road at a highway intersection).

[0302] As shown in Figure 15, when the vehicle is detected to be stationary, the vehicle can display a prompt message on the central control screen: "The vehicle is currently stationary. Abnormal driving behavior has been detected during vehicle movement. Do you wish to start the instructional course?", a cancel control, and a confirm control. In response to the user clicking the confirm control, the vehicle can display video clips related to the abnormal data shown in Figures 13 and 14 (selecting the wrong branch road at the highway exit), along with an analysis and summary of the reasons associated with this abnormal data, on the central control screen.

[0303] For example, as shown in Figure 16, the vehicle can control the central control screen to display video clips related to abnormal data (selecting the wrong branch road at a highway intersection) and mark road signs (e.g., pointing to the road sign with an arrow), and provide text information to analyze and summarize the reasons associated with the abnormal data. For example, the text content is: "After entering the highway, there may be a fork in the road. Observe the information displayed on the road signs. If you need to go to city A, you should drive in the left lane and keep to the left at the fork; if you need to go to city B, you should drive in the right lane and keep to the right at the fork."

[0304] Figures 17-19 show another set of GUIs provided in the embodiments of this application.

[0305] As shown in Figure 17, when the vehicle is detected to be stationary, the vehicle can display a prompt message on the central control screen: "The vehicle is currently stationary. Abnormal driving behavior has been detected during the vehicle's operation. Do you want to start the simulated driving practice?", a cancel control, and a confirm control.

[0306] As shown in Figure 18, in response to the detection that the user has clicked the OK control, the vehicle can control the HUD to display a simulated driving image.

[0307] As shown in Figure 19, in response to receiving user commands for steering wheel, accelerator pedal, brake pedal, or gear shift, the vehicle can control the HUD to display an updated simulated driving image.

[0308] The above example illustrates how the user is prompted with instructional courses and simulated driving exercises inside the vehicle's cabin; however, the embodiments of this application are not limited to this.

[0309] For example, Figure 20 illustrates another GUI provided in an embodiment of this application.

[0310] As shown in Figure 20, the simulated driving image, accelerator pedal control, and brake pedal control are displayed on the tablet's screen. When the tablet is detected to tilt to the left, it indicates that the user is turning the steering wheel to the left; conversely, when the tablet is detected to tilt to the right, it indicates that the user is turning the steering wheel to the right.

[0311] The process of the vehicle control display device displaying the simulated driving image and the control display device displaying the updated simulated driving image based on the operation command can be referred to the description in the above embodiments, and will not be repeated here.

[0312] The embodiments of this application can be applied not only to the field of cockpit teaching, but also to other fields, such as the field of smartphones, smart homes, and industrial remote control.

[0313] a. Smartphone sector

[0314] The multimodal large model technology in this application is not only applicable to in-vehicle teaching systems, but can also be extended to the field of smartphones. For example, it can be applied to augmented reality (AR) education applications, capturing the user's surrounding environment through the phone's camera and combining voice, image, and sensor data to provide personalized learning content to the user in real time. For example, in language learning, users can identify objects or text through the phone's camera and receive real-time learning and feedback through voice, video, or text information generated by the system.

[0315] Technological variations:

[0316] Variations of the instruction understanding unit: Mobile phone users' input can be voice commands, touchscreen interaction, or gesture recognition, which differs from the input methods of in-vehicle systems. Therefore, it can adapt to different input methods and add gesture recognition and touch input processing functions.

[0317] Output module variations: Smartphones primarily output through screen displays and audio feedback. Considering the mobility of mobile users, the output module can be enhanced with support for AR environments, such as overlaying virtual objects or text information onto the screen and combining this with voice feedback to make the learning process more intuitive and immersive.

[0318] b. Smart Home Sector

[0319] In smart home environments, this multimodal big data technology can be used for the safety and health management of family members. For example, through home cameras, microphones, and other sensors, the system can monitor the home environment and the behavior of family members, preventing and analyzing potential dangers. For instance, the system can detect falls or abnormal vital signs in the elderly and issue warnings, immediately notifying family members or medical institutions, while simultaneously providing voice guidance to the elderly on self-rescue measures.

[0320] Technological variations:

[0321] Variations in Sensor Data Preprocessing Units: In smart homes, the types and layouts of sensors differ significantly from those in automotive environments. The system needs to integrate data from various sensors, such as temperature sensors, door and window open / close sensors, and infrared human body sensors, requiring adjustments to the data preprocessing unit to adapt to the diverse inputs of the home environment.

[0322] Variations in output module functionality: In home settings, system feedback can be output through devices such as smart speakers, smart lights, and television screens. For example, when a fire is detected, the system can issue an alarm through the whole-house smart lighting system and display emergency evacuation instructions on the television screen.

[0323] c. Industrial remote control field

[0324] In industrial environments, particularly in remote control and industrial training, it can be applied to the operation training and real-time monitoring of complex equipment. For example, the system can monitor and guide the operator's remote operation behavior in real time through multimodal inputs (such as camera and sensor data), helping novices learn to operate complex machinery in a simulated environment and correct errors in real time.

[0325] Technological variations:

[0326] An evolution of the scene fusion and understanding module: In industrial environments, the objects being operated on typically have higher complexity and more stringent safety requirements. The scene fusion and understanding module needs to enhance its perception of industrial equipment and the operating environment, combining industrial sensors and high-definition cameras to extract key operational data in real time, and simulating the correct and incorrect steps of equipment operation through virtual reality (VR) technology.

[0327] Variations of the simulation generation module: In industrial applications, the simulation generation module can generate high-precision equipment operation simulations, helping operators experience the operation of complex equipment in a virtual environment and providing detailed feedback including equipment status, operating procedures, and safety prompts.

[0328] Similar driver training effects can be achieved by modifying the multimodal information fusion algorithm in this application embodiment, but this application embodiment is not limited to this. For example, different deep learning architectures or more data preprocessing steps may be used to improve the system's recognition accuracy of driver behavior. Alternatively, different modal combinations (such as adding biometric data) may be used to enhance the interactivity and personalization of the system.

[0329] In this embodiment of the application, similar teaching effects can also be achieved through different simulation generation techniques. For example, different generative adversarial networks (GANs) or other generative models can be used to create virtual scenarios and feedback content for driver training, in order to avoid directly infringing on the technical implementation of this invention.

[0330] In this embodiment of the application, similar teaching effects can also be achieved through different simulation generation techniques. For example, different generative adversarial networks (GANs) or other generative models can be used to create virtual scenarios and feedback content for driver training.

[0331] In this embodiment, a similar immersive learning experience can also be achieved by adjusting the hardware configuration or output method of the output module. For example, different types of HUD display technology can be used or a multi-point vibration feedback device can be added inside the vehicle to enhance the feedback effect during the teaching process.

[0332] Multimodal model: In artificial intelligence, a multimodal model is a method that uses multiple different perceptual modalities (such as text, speech, and images) for joint modeling and prediction.

[0333] Generative artificial intelligence (AI) is an AI system capable of generating text, images, or other media in response to prompts. The generating model learns the patterns and structure of input data and then produces new content that is similar to the training data but possesses a degree of novelty, rather than simply classifying or predicting data.

[0334] A convolutional neural network (CNN) is a type of feedforward neural network whose artificial neurons respond to a subset of surrounding units within their coverage area, making it excellent for processing large images. A CNN consists of one or more convolutional layers and a fully connected layer at the top (corresponding to a classic neural network), as well as associated weights and pooling layers. This structure allows CNNs to utilize the two-dimensional structure of the input data. Compared to other deep learning architectures, CNNs deliver better results in image and speech recognition.

[0335] LSTM is a type of time-recurrent neural network. Due to its unique design structure, LSTM is suitable for processing and predicting important events in time series with very long intervals and delays.

[0336] A digital filter is a discrete-time system that filters digital signals to obtain the desired response characteristics.

[0337] The Visual Transformer (ViT) is a deep learning model based on the Transformer architecture for image recognition and computer vision tasks. Unlike traditional Convolutional Neural Networks (CNNs), ViT directly treats the image as a sequential input and utilizes a self-attention mechanism to process pixel relationships within the image. ViT divides the image into a series of patches and converts each patch into a vector representation as the input sequence. These vectors are then processed by a multi-layered Transformer encoder, which incorporates self-attention mechanisms and feedforward neural network layers. This captures the contextual dependencies at different locations within the image. Finally, specific visual tasks can be performed by classifying or regressing the output of the Transformer encoder.

[0338] An inertial measurement unit (IMU) is a device that measures an object's three-axis attitude angles (or angular rates) and acceleration. Typically, an IMU contains three gyroscopes along three axes and accelerometers in three directions to measure the object's angular velocity and acceleration in three-dimensional space, thereby calculating the object's attitude. To improve reliability, more sensors can be added for each axis. Generally, the IMU is mounted at the center of gravity of the object being measured.

[0339] Figure 21 shows a schematic block diagram of a cockpit teaching device 2100 provided in an embodiment of this application. The device 2100 includes: an acquisition unit 2110, configured to acquire first data, including data collected by sensors during vehicle operation and / or images acquired by a vehicle recorder during operation; the acquisition unit 2110 is further configured to acquire first information associated with abnormal data based on the first data, the abnormal data indicating abnormal driving behavior and / or abnormal emotional state of the driver; and a control unit 2120, configured to, when the vehicle is stationary, control a prompting device to prompt the user with teaching content based on the first information.

[0340] Optionally, the control unit 2120 is further configured to: control the display device to display a first simulated driving image based on the first information; and control the display device to display a second simulated driving image based on the user's operation command for at least one of the components, such as the steering wheel, accelerator pedal, or brake pedal.

[0341] Optionally, the control unit 2120 is specifically configured to: control the head-up display (HUD) to display the first simulated driving image based on the first information.

[0342] Optionally, the device 2100 further includes an image generation unit, used to input the operation command and a preset simulated driving image into an image generation model to obtain the second simulated driving image before the control unit controls the display device to display the second simulated driving image.

[0343] Optionally, the image generation unit is specifically used for: sending the operation instruction to the cloud server; and receiving the second simulated driving image sent by the cloud server based on the operation instruction and the preset simulated driving image.

[0344] Optionally, the acquisition unit 2110 is further configured to: acquire a preset simulated driving image based on data from the first data before the abnormal data is generated and data from the time period during which the abnormal data is generated, before the control unit controls the display device to display the first simulated driving image; and acquire the first simulated driving image from the preset simulated driving image.

[0345] Optionally, the control unit 2120 is specifically used to: control the display device to display the second simulated driving image and adjust the feedback parameters of the driver's seat according to the operation command.

[0346] Optionally, the control unit 2120 is specifically used to adjust one or more of the vibration frequency, vibration intensity, and tilt angle of the seat in the driver's area.

[0347] Optionally, the control unit 2120 is specifically configured to: control the prompting device to prompt the user with a teaching course associated with the abnormal data based on the first information; control the prompting device to prompt the user to perform a simulated driving exercise for the abnormal data at the end of the teaching course; and control the display device to display the first simulated driving image in response to receiving input from the user confirming the performance of the simulated driving exercise.

[0348] Optionally, the acquisition unit 2110 is specifically used to: input the first data into the generation model to obtain one or more of the following: timestamp information of the abnormal data, teaching courses associated with the abnormal data, and information of targets associated with the abnormal data.

[0349] Optionally, the control unit 2120 is specifically configured to perform one or more of the following: control the prompting device to indicate that the abnormal data was not generated by the driver's misoperation or was generated by the driver's misoperation; control the display device to display a first image outside the cabin during the time period in which the abnormal data occurred, or control the display device to display a second image outside the cabin during the time period in which the abnormal data occurred and a period before the abnormal data occurred; prompt the user with information about the target while controlling the display device to display the first image or the second image; control the display device to display the driver's emotional state when the abnormal data occurred, or control the display device to display the driver's emotional state during the time period in which the abnormal data occurred and a period before the abnormal data occurred; or control the display device to display the teaching course.

[0350] Figure 22 shows a schematic block diagram of a teaching content generation device 2200 provided in an embodiment of this application. The device 2200 includes: an acquisition unit 2210, configured to acquire first data, which includes data collected by sensors during vehicle operation and / or images acquired by a vehicle recorder during operation; and a teaching content generation unit 2220, configured to input the first data into a generation model to obtain teaching content, which includes one or more of the following: timestamp information of abnormal data, teaching courses associated with the abnormal data, and information about targets associated with the abnormal data. The abnormal data indicates abnormal driving behavior and / or abnormal emotional state of the driver.

[0351] Optionally, the generation model includes a first sub-generation model and a second sub-generation model. The teaching content generation unit 2220 is specifically used to: input the first data into the first sub-generation model to obtain a first generation result, which includes the timestamp information, a summary and / or analysis of the reasons for generating the abnormal data, and the target information; input the first generation result into the second sub-generation model to obtain the teaching course.

[0352] Optionally, the device 2200 further includes a control unit for controlling the vehicle's prompting device to provide prompts to the driver based on the teaching content when the vehicle is stationary.

[0353] Optionally, the control unit is specifically configured to perform one or more of the following: control the prompting device to indicate that the abnormal data was not generated by the driver's misoperation or was generated by the driver's misoperation; control the display device to display a first image outside the cabin during the time period in which the abnormal data occurred, or control the display device to display a second image outside the cabin during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; prompt the user with information about the target while controlling the display device to display the first image or the second image; control the display device to display the driver's emotional state when the abnormal data occurred, or control the display device to display the driver's emotional state during a period before the abnormal data occurred and during the time period in which the abnormal data occurred; or control the display device to display the instructional course.

[0354] Figure 23 shows a schematic block diagram of a simulated driving image generation device 2300 provided in an embodiment of this application. The device 2300 includes: an acquisition unit 2310, configured to acquire first data, which includes data collected by sensors during vehicle operation and / or images acquired by a vehicle recorder during operation; and an image generation unit 2320, configured to input at least a portion of the first data into an image generation model to obtain a preset simulated driving image, which includes simulated driving images associated with abnormal data indicating abnormal driving behavior and / or abnormal emotional state of the driver.

[0355] Optionally, the preset simulated driving image includes a period of time before the abnormal data was generated and environmental information around the vehicle during the period when the abnormal data was generated.

[0356] Optionally, the device further includes: a first control unit for controlling the display device of the terminal device to display the first simulated driving image in the preset simulated driving image.

[0357] Optionally, the terminal device is a vehicle, and the first control unit is specifically used to control the vehicle's HUD to display the first simulated driving image.

[0358] Optionally, the acquisition unit is further configured to acquire user operation instructions for the terminal device;

[0359] The image generation unit is also used to input the operation command and the preset simulated driving image into the image generation model to obtain a second simulated driving image.

[0360] Optionally, the device further includes a second control unit. The image generation unit is also used to input the operation command and the preset simulated driving image into the image generation model to obtain evaluation content or explanation content for the operation command. The second control unit is used to control the prompting device of the terminal device to prompt the user with the evaluation content or explanation content.

[0361] Optionally, the image generation unit 2320 is specifically used to: input the data associated with the abnormal data in the first data into the image generation model to obtain the preset simulated driving image.

[0362] It should be understood that the division of units in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the units in the device can be implemented by a processor calling software; for example, the device includes a processor connected to memory, which stores instructions. The processor calls the instructions stored in memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be, for example, a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. The functions of some or all units can be implemented through the design of the hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all units are implemented through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby implementing the functions of some or all units. All units of the above devices can be implemented entirely through processor calling software, or entirely through hardware circuits, or partially through processor calling software with the remaining parts implemented through hardware circuits.

[0363] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0364] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0365] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together as a System-on-a-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and AI processor, CPU and GPU, etc.

[0366] This application also provides an apparatus comprising a processing unit and a storage unit, wherein the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the apparatus to perform the methods or steps described in the above embodiments.

[0367] Alternatively, if the device is located in a vehicle, the aforementioned processing unit may be the processor 121-12n shown in FIG1.

[0368] This application also provides a cockpit teaching system, which may include a computing platform and a prompting device, the computing platform including the aforementioned device.

[0369] This application also provides a vehicle that may include the above-described device or system.

[0370] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0371] This application also provides a computer-readable medium storing program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0372] This application also provides a chip, which includes a circuit for performing the methods described in the above embodiments.

[0373] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, power-on erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0374] It should be understood that in the embodiments of this application, the memory may include read-only memory and random access memory, and provides instructions and data to the processor.

[0375] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0376] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0377] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0378] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0379] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0380] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0381] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0382] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be covered. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cockpit teaching method, characterized in that, include: Acquire first data, which includes data collected by sensors during vehicle operation and / or images acquired by the vehicle's dashcam during vehicle operation; Based on the first data, obtain first information associated with the abnormal data, wherein the abnormal data indicates abnormal driving behavior of the driver and / or abnormal emotional state of the driver; When the vehicle is stationary, the control prompting device prompts the user with teaching content based on the first information.

2. The method according to claim 1, characterized in that, The method further includes: Based on the first information, the control display device displays the first simulated driving image; The display device is controlled to display a second simulated driving image based on the user's operation command on at least one of the components, namely the steering wheel, accelerator pedal, or brake pedal.

3. The method according to claim 2, characterized in that, The step of controlling the display device to display the first simulated driving image based on the first information includes: Based on the first information, the head-up display (HUD) is controlled to display the first simulated driving image.

4. The method according to claim 2 or 3, characterized in that, Before controlling the display device to display the second simulated driving image based on the user's operation on at least one of the components, such as the steering wheel, accelerator pedal, or brake pedal, the method further includes: The operation instructions and the preset simulated driving image are input into the image generation model to obtain the second simulated driving image.

5. The method according to claim 4, characterized in that, The step of inputting the operation command and the preset simulated driving image into the image generation model to obtain the second simulated driving image includes: Send the operation command to the cloud server; Receive the second simulated driving image sent by the cloud server based on the operation command and the preset simulated driving image.

6. The method according to any one of claims 2 to 5, characterized in that, Before controlling the display device to display the first simulated driving image based on the first information, the method further includes: Based on the data from the period before the abnormal data was generated and the data during the period when the abnormal data was generated in the first data, a preset simulated driving image is obtained; The first simulated driving image is obtained from the preset simulated driving image.

7. The method according to any one of claims 2 to 6, characterized in that, The step of controlling the display device to display a second simulated driving image based on user operation commands to at least one of the steering wheel, accelerator pedal, or brake pedal includes: According to the operation instructions, the display device is controlled to display the second simulated driving image and the feedback parameters of the seat in the driver's area are adjusted.

8. The method according to claim 7, characterized in that, The feedback parameters for adjusting the driver's seat include: Adjust one or more of the vibration frequency, vibration intensity, and tilt angle of the seat in the driver's area.

9. The method according to any one of claims 2 to 8, characterized in that, The step of controlling the prompting device to prompt the user with teaching content based on the first information includes: Based on the first information, the prompting device is controlled to prompt the user with teaching courses associated with the abnormal data; At the end of the teaching course, the prompting device is controlled to prompt the user to perform simulated driving exercises in response to the abnormal data; The control display device displays a first simulated driving image, including: In response to receiving user confirmation of the simulated driving exercise, the display device is controlled to display the first simulated driving image.

10. The method according to any one of claims 1 to 9, characterized in that, The step of obtaining first information associated with the abnormal data based on the first data includes: The first data is input into the generation model to obtain one or more of the following: timestamp information of the abnormal data, teaching courses associated with the abnormal data, and information of targets associated with the abnormal data.

11. The method according to claim 10, characterized in that, The step of controlling the prompting device to prompt the user with teaching content based on the first information includes: Perform one or more of the following: The control prompt device indicates that the abnormal data was not caused by the driver's misoperation or was caused by the driver's misoperation. The control display device displays a first image outside the cockpit during the time period in which the abnormal data was generated, or the control display device displays a second image outside the cockpit during a period before the abnormal data was generated and during the time period in which the abnormal data was generated; When controlling the display device to display the first image or the second image, the user is prompted with information about the target; The display device can be controlled to display the driver's emotional state at the time the abnormal data occurred, or the display device can be controlled to display the driver's emotional state for a period of time before the abnormal data occurred and during the period when the abnormal data occurred; or... Control the display device to display the teaching course.

12. A method for generating teaching content, characterized in that, include: Acquire first data, which includes data collected by sensors during vehicle operation and / or images acquired by the vehicle's dashcam during vehicle operation; The first data is input into the generation model to obtain teaching content, which includes one or more of the following: timestamp information of the abnormal data, teaching courses associated with the abnormal data, and information of targets associated with the abnormal data. The abnormal data indicates abnormal driving behavior of the driver and / or abnormal emotional state of the driver.

13. The method according to claim 12, characterized in that, The generative model includes a first sub-generative model and a second sub-generative model. The step of inputting the first data into the generative model to obtain teaching content includes: The first data is input into the first sub-generation model to obtain a first generation result. The first generation result includes the timestamp information, a summary and / or analysis of the reasons for generating the abnormal data, and the target information. The first generation result is input into the second sub-generation model to obtain the teaching course.

14. The method according to claim 12 or 13, characterized in that, The method further includes: When the vehicle is stationary, the control prompting device provides prompts to the driver based on the teaching content.

15. The method according to claim 14, characterized in that, The step of controlling the prompting device to provide prompts to the driver based on the teaching content includes: Perform one or more of the following: The control prompt device indicates that the abnormal data was not caused by the driver's misoperation or was caused by the driver's misoperation. The control display device displays a first image outside the cockpit during the time period in which the abnormal data was generated, or the control display device displays a second image outside the cockpit during a period before the abnormal data was generated and during the time period in which the abnormal data was generated. When controlling the display device to display the first image or the second image, the user is prompted with information about the target; The display device can be controlled to display the driver's emotional state at the time the abnormal data occurred, or the display device can be controlled to display the driver's emotional state for a period of time before the abnormal data occurred and during the period when the abnormal data occurred; or... Control the display device to display the teaching course.

16. A method for generating simulated driving images, characterized in that, include: Acquire first data, which includes data collected by sensors during vehicle operation and / or images acquired by the vehicle's dashcam during vehicle operation; At least a portion of the data in the first data is input into an image generation model to obtain a preset simulated driving image. The preset simulated driving image includes a preset simulated driving image associated with abnormal data, wherein the abnormal data indicates abnormal driving behavior of the driver and / or abnormal emotional state of the driver.

17. The method according to claim 16, characterized in that, The preset simulated driving image includes a period of time before the abnormal data is generated and environmental information around the vehicle during the period when the abnormal data is generated.

18. The method according to claim 16 or 17, characterized in that, The method further includes: The display device of the control terminal equipment displays the first simulated driving image from the preset simulated driving images.

19. The method according to claim 18, characterized in that, The terminal device is a vehicle, and the display device of the control terminal device displays the first simulated driving image from the preset simulated driving images, including: Control the HUD to display the first simulated driving image.

20. The method according to any one of claims 16 to 19, characterized in that, The method further includes: Obtain user operation commands for the terminal device; The operation instructions and the preset simulated driving image are input into the image generation model to obtain the second simulated driving image.

21. The method according to claim 20, characterized in that, The method further includes: The operation instructions and the preset simulated driving image are input into the image generation model to obtain evaluation or explanation content for the operation instructions; The prompting device of the terminal device prompts the user with the evaluation content or explanation content.

22. The method according to any one of claims 16 to 21, characterized in that, The step of inputting at least a portion of the data from the first data into the generation model to obtain a simulated driving image includes: The data associated with the abnormal data in the first data is input into the image generation model to obtain the preset simulated driving image.

23. A cockpit teaching device, characterized in that, include: An acquisition unit is configured to acquire first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by a vehicle recorder during vehicle operation; The acquisition unit is further configured to acquire first information associated with the abnormal data based on the first data, wherein the abnormal data indicates abnormal driving behavior of the driver and / or abnormal emotional state of the driver. The control unit is configured to, when the vehicle is stationary, control the prompting device to provide instructional content to the user based on the first information.

24. The apparatus according to claim 23, characterized in that, The control unit is also used for: Based on the first information, the control display device displays the first simulated driving image; The display device is controlled to display a second simulated driving image based on the user's operation command on at least one of the components, namely the steering wheel, accelerator pedal, or brake pedal.

25. The apparatus according to claim 24, characterized in that, The control unit is specifically used for: Based on the first information, the head-up display (HUD) is controlled to display the first simulated driving image.

26. The apparatus according to claim 24 or 25, characterized in that, The device further includes: An image generation unit is used to input the operation command and a preset simulated driving image into an image generation model before the control unit controls the display device to display the second simulated driving image, so as to obtain the second simulated driving image.

27. The apparatus according to claim 26, characterized in that, The image generation unit is specifically used for: Send the operation command to the cloud server; Receive the second simulated driving image sent by the cloud server based on the operation command and the preset simulated driving image.

28. The apparatus according to any one of claims 24 to 27, characterized in that, The acquisition unit is further configured to: Before the control unit controls the display device to display the first simulated driving image, a preset simulated driving image is obtained based on the data from the first data, which is obtained during a period of time before the abnormal data is generated and during the period of time when the abnormal data is generated. The first simulated driving image is obtained from the preset simulated driving image.

29. The apparatus according to any one of claims 24 to 28, characterized in that, The control unit is specifically used for: According to the operation instructions, the display device is controlled to display the second simulated driving image and the feedback parameters of the seat in the driver's area are adjusted.

30. The apparatus according to claim 29, characterized in that, The control unit is specifically used for: Adjust one or more of the vibration frequency, vibration intensity, and tilt angle of the seat in the driver's area.

31. The apparatus according to any one of claims 24 to 30, characterized in that, The control unit is specifically used for: Based on the first information, the prompting device is controlled to prompt the user with teaching courses associated with the abnormal data; At the end of the teaching course, the prompting device is controlled to prompt the user to perform simulated driving exercises in response to the abnormal data; In response to receiving user confirmation of the simulated driving exercise, the display device is controlled to display the first simulated driving image.

32. The apparatus according to any one of claims 23 to 31, characterized in that, The acquisition unit is specifically used for: The first data is input into the generation model to obtain one or more of the following: timestamp information of the abnormal data, teaching courses associated with the abnormal data, and information of targets associated with the abnormal data.

33. The apparatus according to claim 32, characterized in that, The control unit is specifically configured to perform one or more of the following: The control prompt device indicates that the abnormal data was not caused by the driver's misoperation or was caused by the driver's misoperation. The control display device displays a first image outside the cockpit during the time period in which the abnormal data was generated, or the control display device displays a second image outside the cockpit during a period before the abnormal data was generated and during the time period in which the abnormal data was generated; When controlling the display device to display the first image or the second image, the user is prompted with information about the target; The display device is controlled to display the driver's emotional state when the abnormal data is generated, or the display device is controlled to display the driver's emotional state for a period of time before the abnormal data is generated and during the period when the abnormal data is generated. or, Control the display device to display the teaching course.

34. A device for generating teaching content, characterized in that, include: An acquisition unit is configured to acquire first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by a vehicle recorder during vehicle operation; The teaching content generation unit is used to input the first data into the generation model to obtain teaching content. The teaching content includes one or more of the following: timestamp information of the abnormal data, teaching courses associated with the abnormal data, and information of targets associated with the abnormal data. The abnormal data indicates abnormal driving behavior of the driver and / or abnormal emotional state of the driver.

35. The apparatus according to claim 34, characterized in that, The generative model includes a first sub-generative model and a second sub-generative model. The teaching content generation unit is specifically used for: The first data is input into the first sub-generation model to obtain a first generation result. The first generation result includes the timestamp information, a summary and / or analysis of the reasons for generating the abnormal data, and the target information. The first generation result is input into the second sub-generation model to obtain the teaching course.

36. The apparatus according to claim 34 or 35, characterized in that, The device further includes: The control unit is used to control the vehicle's prompting device to provide prompts to the driver based on the teaching content when the vehicle is stationary.

37. The apparatus according to claim 36, characterized in that, The control unit is specifically configured to perform one or more of the following: The control prompt device indicates that the abnormal data was not caused by the driver's misoperation or was caused by the driver's misoperation. The control display device displays a first image outside the cockpit during the time period in which the abnormal data was generated, or the control display device displays a second image outside the cockpit during a period before the abnormal data was generated and during the time period in which the abnormal data was generated. When controlling the display device to display the first image or the second image, the user is prompted with information about the target; The display device is controlled to display the driver's emotional state when the abnormal data is generated, or the display device is controlled to display the driver's emotional state for a period of time before the abnormal data is generated and during the period when the abnormal data is generated. or, Control the display device to display the teaching course.

38. An apparatus for generating simulated driving images, characterized in that, include: An acquisition unit is configured to acquire first data, the first data including data collected by sensors during vehicle operation, and / or images acquired by a vehicle recorder during vehicle operation; An image generation unit is configured to input at least a portion of the data in the first data into an image generation model to obtain a preset simulated driving image. The preset simulated driving image includes a simulated driving image associated with abnormal data, wherein the abnormal data indicates abnormal driving behavior of the driver and / or abnormal emotional state of the driver.

39. The apparatus according to claim 38, characterized in that, The preset simulated driving image includes a period of time before the abnormal data is generated and environmental information around the vehicle during the period when the abnormal data is generated.

40. The apparatus according to claim 38 or 39, characterized in that, The device further includes: The first control unit is used to control the display device of the terminal device to display the first simulated driving image in the preset simulated driving image.

41. The apparatus according to claim 40, characterized in that, The terminal device is a vehicle, and the first control unit is specifically used for: Control the HUD to display the first simulated driving image.

42. The apparatus according to any one of claims 38 to 41, characterized in that, The acquisition unit is further configured to acquire user operation instructions for the terminal device; The image generation unit is further configured to input the operation command and the preset simulated driving image into the image generation model to obtain a second simulated driving image.

43. The apparatus according to claim 42, characterized in that, The device also includes a second control unit. The image generation unit is also used to input the operation command and the preset simulated driving image into the image generation model to obtain evaluation content or explanation content for the operation command; The second control unit is used to control the prompting device of the terminal device to prompt the user with the evaluation content or explanation content.

44. The apparatus according to any one of claims 38 to 43, characterized in that, The image generation unit is specifically used for: The data associated with the abnormal data in the first data is input into the image generation model to obtain the preset simulated driving image.

45. A cockpit teaching device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to cause the apparatus to perform the method as described in any one of claims 1 to 11.

46. ​​A device for generating teaching content, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to cause the apparatus to perform the method as described in any one of claims 12 to 15.

47. An apparatus for generating simulated driving images, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to cause the apparatus to perform the method as described in any one of claims 16 to 22.

48. A cockpit teaching system, characterized in that, It includes a prompting device and a computing platform, wherein the computing platform includes the device as described in any one of claims 23 to 33 and 45.

49. A vehicle, characterized in that, Includes the apparatus as described in any one of claims 23 to 33 and 45, or the system as described in claim 48.

50. A cloud server, characterized in that, Includes the apparatus as described in any one of claims 34 to 44, 46, and 47.

51. A computer-readable storage medium, characterized in that, It stores instructions that, when executed by a processor, cause the processor to implement the method as described in any one of claims 1 to 22.

52. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 22.

53. A chip, characterized in that, The chip includes circuitry for performing the method as described in any one of claims 1 to 22.

Images