Vehicle control method, storage medium, program product, device, system and vehicle

Abstract

The invention relates to a vehicle control method, a storage medium, a program product, equipment, a system and a vehicle. The method comprises the following steps: controlling a vehicle to provide full-cabin service according to at least one group of object state data; wherein the at least one group of object state data is matched with the object distribution position of the driving object in the vehicle. According to the method and the device, invalid object state data can be prevented from being acquired from an area without a driving object in the vehicle cabin, the invalid object state data is further prevented from being stored and processed, and the computing power and the memory bandwidth required by the whole cabin service are reduced. In addition, full-cabin services can be provided for the driving and riding objects in the vehicle cabin in a targeted mode, and the accuracy of the full-cabin services is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a vehicle control method, storage medium, program product, device, system and vehicle. Background Art

[0002] With the advancement of vehicle intelligence and the widespread application of electronic technology, users are increasingly demanding full-cabin services. These services can be applied in areas such as intelligent assisted driving and intelligent cockpits, providing users with a more enjoyable, enriching, and engaging interactive experience and driving experience. Providing full-cabin services is a technical challenge currently being researched by the industry. Summary of the Invention

[0003] The embodiments of the present application provide a vehicle control method, storage medium, program product, device, system and vehicle, which can reduce the computing power and memory bandwidth required for full-cabin service and improve the accuracy of full-cabin service, so as to at least partially solve the above-mentioned technical problems.

[0004] In order to achieve the above-mentioned purpose, according to the first aspect of the present application, a vehicle control system is provided, which includes: a domain controller for controlling the vehicle to provide full-cabin service based on at least one set of object status data; wherein, at least one set of the object status data matches the object distribution position of the driving objects in the vehicle.

[0005] Optionally, the domain controller includes at least one of the following: a cockpit domain controller and a cabin-cockpit integrated central computing domain controller.

[0006] Optionally, the vehicle control system further includes: a data acquisition system connected to the domain controller, for acquiring data required for the full-cabin service.

[0007] Optionally, the data acquisition system includes: a control signal acquisition module, used to obtain a motion control signal of the vehicle; wherein the motion control signal is used to determine motion characteristic data of the vehicle.

[0008] Optionally, the motion control signal includes at least one of the following: a body domain control signal, a power domain control signal, and an energy domain control signal.

[0009] Optionally, the data acquisition system includes: an off-vehicle acquisition module, configured to acquire off-vehicle environmental data of the vehicle; wherein the off-vehicle environmental data is used to determine environmental characteristic data of the vehicle.

[0010] Optionally, the off-vehicle acquisition module includes at least one of the following: an off-vehicle camera unit, an off-vehicle sensor unit.

[0011] Optionally, the external camera unit includes at least one of the following: a front camera of the vehicle body, a rear camera of the vehicle body, a left camera of the vehicle body, a right camera of the vehicle body, and a face recognition camera.

[0012] Optionally, the off-vehicle sensor unit includes at least one of the following: a temperature sensor, a humidity sensor, and an air pressure sensor.

[0013] Optionally, the data acquisition system includes: an in-vehicle acquisition module, used to obtain the in-vehicle environment data of the vehicle; wherein the in-vehicle environment data is used to determine at least one of the object status data, the object distribution position of the driving object, and the environmental characteristic data of the vehicle.

[0014] Optionally, the in-vehicle acquisition module includes at least one of the following: an in-vehicle camera unit, an in-vehicle sensor unit.

[0015] Optionally, the in-vehicle camera unit includes at least one of the following: DMS, OMS, IMS, and iris recognizer.

[0016] Optionally, the in-vehicle sensor unit includes at least one of the following: an in-vehicle radar, an IMU, and an occupancy detector.

[0017] Optionally, the data acquisition system includes: a navigation and positioning module, used to obtain navigation and positioning data of the vehicle.

[0018] Optionally, the navigation and positioning module includes at least one of the following: GPS, Beidou, wireless base station positioning unit, and IMU.

[0019] Optionally, the data acquisition system includes: an acoustic module for processing sound data; wherein the sound data is used to determine interactive control data of the vehicle or to provide the full-cabin service.

[0020] Optionally, the acoustic module includes at least one of the following: a microphone, a speaker, an amplifier, a vibration motor, and A2B.

[0021] Optionally, the vehicle control system further includes: a communication system connected to the domain controller and the data acquisition system, and configured to transmit data required for the full-cabin service between the domain controller and the data acquisition system.

[0022] Optionally, the communication system includes at least one of the following: a wireless communication module and a wired communication module.

[0023] Optionally, the wireless communication module includes at least one of the following: 4G, 5G, WiFi, BT, NFC, and UWB.

[0024] Optionally, the wired communication module includes at least one of the following: Ethernet, GMSL, FPD-LINK, CAN, CAN-FD, and LIN.

[0025] According to a second aspect of the present application, a vehicle control method is provided, the method comprising: controlling a vehicle to provide full-cabin service based on at least one set of object status data; wherein at least one set of the object status data matches the object distribution position of the driving objects in the vehicle.

[0026] Optionally, the object status data includes at least one of the following: an image of the object riding, voice data of the object, text data of the object, body temperature data of the object, pulse data of the object, and respiratory rate of the object.

[0027] Optionally, the method further includes: identifying object distribution positions of objects driven in the vehicle; and acquiring at least one set of object status data based on the object distribution positions.

[0028] Optionally, the identifying the object distribution positions of the driving objects in the vehicle includes: identifying the object distribution positions of the driving objects in the vehicle based on a panoramic image of the vehicle cabin.

[0029] Optionally, obtaining at least one set of object status data based on the object distribution position includes: obtaining at least one set of object status data that matches the object distribution position from multiple sets of object status data; wherein each set of object status data in the multiple sets of object status data corresponds to at least one seat position of the vehicle.

[0030] Optionally, controlling the vehicle to provide full-cabin service based on at least one set of object status data includes: inputting each set of the object status data into a first model so that the first model outputs object perception data; and controlling the vehicle to provide full-cabin service based on the object perception data corresponding to at least one set of the object status data.

[0031] Optionally, inputting each group of the object state data into the first model so that the first model outputs object perception data includes: inputting each group of the object state data into the embedding module of the first model so that the embedding module outputs object feature data; inputting the object feature data corresponding to each group of the object state data into the filling module of the first model so that the filling module outputs starting prediction data; and inputting the starting prediction data corresponding to each group of the object state data into the decoding module of the first model so that the decoding module outputs object perception data.

[0032] Optionally, controlling the vehicle to provide full-cabin service based on at least one set of object status data includes: controlling the vehicle to provide full-cabin service based on at least one set of object status data and driving status data of the vehicle.

[0033] Optionally, the driving status data includes at least one of the following: motion characteristic data, environmental characteristic data, navigation positioning data, and interactive control data.

[0034] Optionally, the motion characteristic data includes at least one of the following: vehicle body posture, vehicle speed, and operating status.

[0035] Optionally, the environmental characteristic data includes at least one of the following: road data, weather data, obstacle data, pedestrian data, and temperature.

[0036] Optionally, the navigation positioning data includes at least one of the following: vehicle geographic location, navigation route, gas station data, service area data, and building data.

[0037] Optionally, the interactive control data includes at least one of the following: sound data, gesture data, and touch data.

[0038] Optionally, controlling the vehicle to provide full-cabin service based on at least one set of object status data includes: controlling the vehicle to provide full-cabin service based on at least one set of object status data when a first condition is met.

[0039] Optionally, the first condition includes at least one of the following: the vehicle is in motion and an enabling instruction for the full-cabin service is received.

[0040] Optionally, the method further includes: stopping the full-cabin service when a second condition is met.

[0041] Optionally, the second condition includes at least one of the following: receiving a stop instruction for the full-cabin service, and the vehicle being stationary.

[0042] Optionally, the method further includes: when a third condition is met, controlling a data acquisition system of the vehicle to enter a standby state; wherein the data acquisition system is used to obtain data required for the full-cabin service.

[0043] Optionally, the third condition includes: the domain controller of the vehicle completes startup.

[0044] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the vehicle control method as described above is implemented.

[0045] According to a fourth aspect of the present application, a computer program product is provided, comprising a computer program or instructions, which implement the vehicle control method as described above when executed by a processor.

[0046] According to a fifth aspect of the present application, an electronic device is provided, comprising: a memory on which a computer program or instruction is stored; and a processor for executing the computer program or instruction in the memory to implement the vehicle control method as described above.

[0047] According to a sixth aspect of the present application, a vehicle is provided, comprising the vehicle control system as described above, or comprising the electronic device as described above.

[0048] In summary, the technical solution provided by the embodiment of the present application controls the vehicle to provide full-cabin service based on at least one set of object status data. Among them, at least one set of object status data matches the object distribution position of the driving objects in the vehicle. The embodiment of the present application obtains matching object status data based on the object distribution position of the driving objects in the vehicle cabin, which can avoid obtaining invalid object status data for areas in the vehicle cabin where there are no driving objects, further avoids the storage and processing of invalid object status data, and reduces the computing power and memory bandwidth required for full-cabin service. In addition, the embodiment of the present application provides full-cabin service based on object status data that matches the object distribution position. The basis of the full-cabin service is the object status data that matches the actual object distribution position, and can provide targeted full-cabin service for the driving objects in the vehicle cabin, thereby improving the accuracy of the full-cabin service.

[0049] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0051] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same drawing numbers represent the same parts in the following description.

[0052] Figure 1 is a schematic diagram of a vehicle control system provided in an embodiment of the present application;

[0053] Figure 2 is a flow chart of a vehicle control method provided by an embodiment of the present application;

[0054] Figure 3 is a flow chart of another vehicle control method provided by an embodiment of the present application;

[0055] Figure 4 is a flow chart of another vehicle control method provided by an embodiment of the present application;

[0056] Figure 5 is a flow chart of another vehicle control method provided by an embodiment of the present application;

[0057] Figure 6 It is a schematic diagram of a vehicle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0059] With the advancement of vehicle intelligence and the widespread application of electronic technology in the automotive field, users are increasingly demanding intelligent cockpit services. In particular, with the advancement of chip technology and the continuous improvement of platform computing power, the application of large-scale AI (artificial intelligence) models on the edge has gradually become a focus of competition among major vehicle manufacturers.

[0060] The application of AI big models in vehicles is mainly reflected in the two areas of intelligent assisted driving and smart cockpits. In the field of intelligent assisted driving, by collecting various types of sensor data and converting them into an interactive 3D simulation environment, it is used to enrich the driver's perception and provide more comprehensive driving environment information to achieve a safer and more enjoyable driving experience. In the field of smart cockpits, through various perception systems inside and outside the cabin, and based on the integration of AI big models and multimodal interaction functions, users are provided with richer and more considerate services. At the same time, through deeper interaction with users in the car in terms of language, behavior, visual perception, emotion capture, etc., through AI big models, multimodal perception, and big data processing, users are provided with a more intelligent, personalized, and considerate full-scene interactive experience and driving experience.

[0061] With the development of Transformer technology and its mature application in various large-scale model fields, the thinking capabilities of large-scale models on the edge of intelligent cockpits are constantly improving. The application of AI large-scale models in cockpits will no longer be limited to simple knowledge Q&A, scene recommendations, music playback, etc., but will evolve towards intelligent agents and multimodal integration, integrating the interaction between people, vehicles, and the environment, and completing user-assigned tasks based on user preferences, emotions, and feelings. This will provide users with more intuitive, comprehensive, and caring services, thereby strengthening user-vehicle stickiness. Therefore, the application of AI multimodal reasoning large-scale models in vehicles has become a key technical development direction for major automakers, AI algorithm manufacturers, and application solution providers.

[0062] Although large AI multimodal reasoning models have a wide range of application scenarios, their application in vehicles also faces a prominent technical challenge: limited end-to-end computing power and memory bandwidth. These large AI multimodal reasoning models must process sensory data from various vehicle sensors in real time, such as cameras, voice, vehicle control, and maps. They must also exchange and store data between various multi-core heterogeneous processors, such as the CPU (Central Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), MCU (Microcontroller Unit), cache, and DDR (Double Data Rate Synchronous Dynamic Random-Access Memory). Therefore, deploying large AI multimodal reasoning models in vehicles presents significant challenges in terms of computing power, memory bandwidth, and capacity requirements of the end-to-end computing platform.

[0063] In related technologies, large models on the end are mainly used for voice interaction and have small model parameters (such as within 1 billion). However, users want to provide more comprehensive, richer, more intelligent and personalized services. Therefore, there is a great demand for model parameters and model thinking ability. Based on this, the computing power and memory bandwidth optimization of large AI multimodal reasoning models are one of the key factors in whether the vehicle can carry the application of larger parameter models.

[0064] In view of this, embodiments of the present application provide a vehicle control method, storage medium, program product, device, system, and vehicle. Embodiments of the present application propose a new multimodal fusion system solution for large AI multimodal reasoning models in intelligent cockpits. Based on this solution and based on the requirements of vehicle multimodal fusion scenarios, a method for optimizing the memory bandwidth of large AI multimodal reasoning models is proposed.

[0065] According to a first aspect of the present application, an embodiment of the present application provides a vehicle control system.

[0066] See also Figure 1 , Figure 1 Schematic diagram of a vehicle control system provided by an embodiment of the present application. Figure 1 As shown, the vehicle control system includes a domain controller 100 .

[0067] The domain controller 100 possesses computing and data processing capabilities. A large AI multimodal reasoning model can be deployed within the domain controller 100 to provide full-cabin services to the vehicle's passengers. In some embodiments, the domain controller 100 includes, but is not limited to, at least one of a cockpit domain controller and a central computing domain controller for the cabin and driver. Providing full-cabin services to passengers using the vehicle's existing domain controllers allows compatibility with existing vehicle designs. In addition to computing and data processing capabilities, the domain controller 100 may also possess storage capacity, peripheral access capabilities, and various sensing capabilities.

[0068] It should be understood that in the embodiments of the present application, the driving objects include but are not limited to at least one of the drivers, passengers, animals, etc.; the full cabin services include but are not limited to at least one of the following services: reminders of the gas station ahead, navigation route reminders, fatigue driving reminders, entertainment playback, restaurant and coffee shop push notifications, emotional value provision, communication with the driving objects, etc. In addition, the AI ​​multimodal reasoning large model can be a single large model that can be used to process multimodal data, or it can include multiple models for processing one or more modal data. Among them, modality refers to the type of data, such as image, voice, text, point cloud, etc.

[0069] The domain controller 100 can be configured to control the vehicle to provide full-cabin service based on at least one set of object status data. The at least one set of object status data matches the object distribution location of passengers in the vehicle. For further information on the various steps performed by the domain controller 100 and their beneficial effects, please refer to the following embodiments and will not be elaborated upon here.

[0070] In some embodiments, as Figure 1As shown, the vehicle control system also includes a data acquisition system 200. Data acquisition system 200 is connected to domain controller 100 to acquire data required for full-cabin service. Data collected by data acquisition system 200 is then transmitted to domain controller 100 for processing to provide full-cabin service, helping to improve the flexibility and comprehensiveness of data collection.

[0071] In some embodiments, as Figure 1 As shown, the data acquisition system 200 includes a control signal acquisition module 210. The control signal acquisition module 210 is used to obtain the motion control signal of the vehicle. The motion control signal is used to determine the motion characteristic data of the vehicle. For other introductions to the motion characteristic data, please refer to the following embodiments, which will not be elaborated here. The control signal acquisition module 210 can be based on the vehicle's electronic and electrical architecture EEA (Electronic Electrical Architecture). In some embodiments, the motion control signal collected by the control signal acquisition module 210 includes at least one of the following: a body domain control signal, a power domain control signal, and an energy domain control signal. By collecting multi-dimensional motion control signals, the vehicle's motion characteristics can be reflected more intuitively and comprehensively, which helps to improve the accuracy of full-cabin service.

[0072] In some embodiments, as Figure 1 As shown, the data acquisition system 200 includes an external vehicle acquisition module 220. The external vehicle acquisition module 220 is used to acquire data about the vehicle's external environment. This external vehicle environment data is used to determine the vehicle's environmental characteristics. For further information on environmental characteristics data, please refer to the following embodiments and will not be elaborated upon here. To achieve more flexible and comprehensive collection of external vehicle environment data, in some embodiments, the external vehicle acquisition module 220 includes at least one of the following: an external vehicle camera unit 221 and an external vehicle sensor unit 222. The specific configurations of the external vehicle camera unit 221 and the external vehicle sensor unit 222 are not limited in this embodiment of the present application; in actual applications, these configurations can be flexibly adjusted based on actual needs. In some embodiments, the external vehicle camera unit 221 includes at least one of the following: a front vehicle camera, a rear vehicle camera, a left vehicle camera, a right vehicle camera, and a facial recognition camera (FaceID camera); and the external vehicle sensor unit 222 includes at least one of the following: a temperature sensor, a humidity sensor, and an air pressure sensor. By collecting multi-dimensional external vehicle environment data, the vehicle's external environment characteristics can be more intuitively and comprehensively reflected, helping to improve the accuracy of full-cabin service.

[0073] In some embodiments, as Figure 1As shown, the data acquisition system 200 includes an in-vehicle acquisition module 230. The in-vehicle acquisition module 230 is used to obtain in-vehicle environmental data of the vehicle. The in-vehicle environmental data is used to determine at least one of object status data, object distribution location of the driving object, and environmental characteristic data of the vehicle. For further description of object status data, object distribution location, and environmental characteristic data, please refer to the following embodiments and will not be repeated here. To achieve more flexible and comprehensive collection of in-vehicle environmental data, in some embodiments, the in-vehicle acquisition module 230 includes at least one of the following: an in-vehicle camera unit 231 and an in-vehicle sensor unit 232. The embodiments of this application do not limit the specific configuration of the in-vehicle camera unit 231 and the in-vehicle sensor unit 232. In actual applications, they can be flexibly configured based on actual needs. In some embodiments, the in-vehicle camera unit 231 includes at least one of the following: a DMS (Driver Monitoring System), an OMS (Occupant Monitoring System), an IMS (Interior Monitoring System), and an iris recognizer; the in-vehicle sensor unit 232 includes at least one of the following: an in-vehicle radar, an IMU (Inertial Measurement Unit), and an occupancy detector. By collecting multi-dimensional in-vehicle environmental data, the characteristics of the vehicle's in-vehicle environment, as well as the status and distribution of the driver and passengers, can be more intuitively and comprehensively reflected, helping to improve the accuracy of full-cabin service.

[0074] In some embodiments, as Figure 1 As shown, the data acquisition system 200 includes a navigation and positioning module 240. The navigation and positioning module 240 is used to obtain navigation and positioning data of the vehicle. The embodiment of the present application does not limit the specific positioning method of the navigation and positioning module 240. In actual application, it can be flexibly set according to needs. In some embodiments, the navigation and positioning module 240 includes at least one of the following: GPS (Global Positioning System), Beidou, wireless base station positioning unit, IMU. Collecting navigation and positioning data through multiple methods can improve the effectiveness and accuracy of navigation and positioning data, which helps to improve the accuracy of full-cabin service.

[0075] In some embodiments, as Figure 1As shown, the data acquisition system 200 includes an acoustic module 250. The acoustic module 250 is used to process sound data. Among them, the sound data is used to determine the interactive control data of the vehicle or to provide full-cabin services. For other introductions to interactive control data, please refer to the following embodiments, which will not be elaborated here. The embodiment of the present application does not limit the specific sound processing method of the acoustic module 250. In actual application, it can be flexibly set according to needs. In some embodiments, the acoustic module 250 includes at least one of the following: microphone, speaker, power amplifier, vibration motor, A2B (Audio Bus). By processing sound data in a variety of ways, not only can the full-cabin service methods be enriched, but also the accuracy of the full-cabin service can be improved.

[0076] To achieve communication between the domain controller 100 and the data acquisition system 200, in some embodiments, such as Figure 1 As shown, the vehicle control system also includes a communication system 300. The communication system 300 is connected to the domain controller 100 and the data acquisition system 200 and is used to transmit data required for full-cabin service between the domain controller 100 and the data acquisition system 200. For example, the domain controller 100 obtains data collected by the data acquisition system 200 through the communication system 300.

[0077] In some embodiments, the communication system 300 includes at least one of the following: a wireless communication module 310 and a wired communication module 320. By combining multiple communication methods, the reliability of data transmission between the domain controller 100 and the data acquisition system 200 can be improved. The present embodiment does not limit the specific network type of the wireless communication module 310 and the wired communication module 320; in actual applications, they can be flexibly configured based on actual needs. In some embodiments, the wireless communication module 310 may include at least one of the following: 4G (The 4th Generation of Mobile Communication Technology), 5G (The 5th Generation of Mobile Communication Technology), WiFi (Wireless Fidelity), BT (Bluetooth), NFC (Near Field Communication), and UWB (Ultra Wideband); the wired communication module 320 may include at least one of the following: Ethernet, GMSL (Gigabit Multimedia Serial Link), FPD-LINK (Flat Panel Display Link), CAN (Controller Area Network), CAN-FD (Controller Area Network-Flexible Data), and LIN (Local Interconnect Network).

[0078] It should be understood that, in addition to implementing communication functions, the communication system 300 can also implement other functions, such as cloud interconnection, vehicle positioning, and identity recognition. Furthermore, in addition to transmitting data between the domain controller 100 and the data acquisition system 200, the communication system 300 can also be used for communication between various modules in the data acquisition system 200, although this embodiment of the present application is not limited to this.

[0079] According to a second aspect of the present application, an embodiment of the present application provides a vehicle control method.

[0080] See also Figure 2 , Figure 2 This is a flow chart of a vehicle control method provided by an embodiment of the present application. The vehicle control method can be applied to the above-mentioned vehicle control system. Figure 2 As shown, the vehicle control method may include the following steps:

[0081] Step S100: controlling the vehicle to provide full-cabin service according to at least one set of object status data; wherein the at least one set of object status data matches the object distribution position of the driving objects in the vehicle.

[0082] A passenger object refers to an object located within the vehicle cabin, including but not limited to at least one of the following: a driver, a passenger, an animal, etc. An object distribution location refers to the distribution location of the passenger object within the vehicle cabin. In embodiments of the present application, the passenger object may include one or more objects. If the passenger object includes one object, the object distribution location may include location data for one object within the vehicle cabin; if the passenger object includes multiple objects, the object distribution location may include location data for multiple objects within the vehicle cabin.

[0083] In an embodiment of the present application, a vehicle is controlled to provide full-cabin service based on at least one set of object status data that matches the object distribution location. The object status data is used to indicate the status of the driving object, including but not limited to at least one of an image, respiratory rate, pulse, body temperature, etc. In an embodiment of the present application, a set of object status data is used to indicate the status of a driving object, and a set of object status data may include at least one type of status data, such as an image, respiratory rate, pulse, body temperature, etc. The object status data matches the object distribution location, which means that the object status data is used to indicate the status of the driving object corresponding to the corresponding position data in the object distribution location.

[0084] Exemplarily, in the case where the driving object includes only the driver, the object distribution position includes the main driving position of the cabin, and the object status data includes a set of data for indicating the status of the driver at the main driving position of the cabin; in the case where the driving object includes the driver and the front passenger seat, the object distribution position includes the main driving position of the cabin and the front passenger seat, and the object status data includes a set of data for indicating the status of the driver at the main driving position of the cabin and a set of data for indicating the status of the occupant at the front passenger seat.

[0085] The vehicle is controlled to provide full-cabin services based on at least one set of object status data. Full-cabin services include, but are not limited to, at least one of the following: upcoming gas station reminders, navigation route reminders, fatigue driving reminders, entertainment playback, restaurant and coffee shop push notifications, emotional value provision, and communication with the driver and passengers. In addition to at least one set of object status data, embodiments of the present application may also combine other data to provide full-cabin services, such as vehicle driving status data. For details, please refer to the following embodiments, which will not be elaborated here. Furthermore, embodiments of the present application may utilize a large AI multimodal reasoning model to perform data calculation and processing to provide full-cabin services.

[0086] In summary, the technical solution provided by the embodiment of the present application controls the vehicle to provide full-cabin service based on at least one set of object status data. Among them, at least one set of object status data matches the object distribution position of the driving objects in the vehicle. The embodiment of the present application obtains matching object status data based on the object distribution position of the driving objects in the vehicle cabin, which can avoid obtaining invalid object status data for areas in the vehicle cabin where there are no driving objects, further avoids the storage and processing of invalid object status data, and reduces the computing power and memory bandwidth required for full-cabin service. In addition, the embodiment of the present application provides full-cabin service based on object status data that matches the object distribution position. The basis of the full-cabin service is the object status data that matches the actual object distribution position, and can provide targeted full-cabin service for the driving objects in the vehicle cabin, thereby improving the accuracy of the full-cabin service.

[0087] In some embodiments, the subject status data includes at least one of the following: subject riding image, subject voice data, subject text data, subject body temperature data, subject pulse data, and subject breathing rate.

[0088] The object riding image refers to the riding image of the driving object in the vehicle cabin. The embodiments of the present application do not limit the specific content of the object riding image. In some embodiments, the object riding image can be a global image, a half-body image, or a facial image of the driving object. For example, a camera is provided at a specific position in front of each seat in the cabin, which can capture the object riding image of the driving object. The acquisition method of the object riding image is simple and fast, which is conducive to the efficient acquisition of the object status data; and the object riding image has rich semantic information, which can objectively and accurately reflect the status of the driver and passengers, improve the accuracy of the object status data, and thus improve the accuracy of the full-cabin service.

[0089] It should be understood from the above embodiments that the subject status data may also include other data, such as subject voice data, subject text data, subject pulse data, subject respiratory rate, subject temperature data, etc. For example, the subject's body temperature may be monitored using an infrared sensor to obtain subject temperature data, while the subject's respiratory rate may be monitored using a radar to obtain subject respiratory rate data, etc.

[0090] In some embodiments, the vehicle control method further includes the following steps:

[0091] Step S010: identifying the object distribution position of the driving object in the vehicle;

[0092] Step S020: Acquire at least one set of object status data according to the object distribution locations.

[0093] In order to obtain object status data that matches the object distribution position, it is necessary to identify the object distribution position of the driving object in the vehicle. The embodiment of the present application does not limit the timing of identifying the object distribution position, and it can be flexibly set in combination with the needs in actual application. In some embodiments, taking into account complex driving scenarios, the vehicle may get on and off the driving object midway. In order to ensure that the full-cabin service is adaptively adjusted to the changes in the driving object, the object distribution position of the driving object in the vehicle can be automatically identified after each door is closed, or the object distribution position can be identified once every preset time after the vehicle is started. Of course, the identification of the object distribution position can also be enabled based on the enabling operation of the driving object.

[0094] To quickly and accurately identify the distribution location of objects, in some embodiments, step S010 may include identifying the distribution location of objects within the vehicle based on a panoramic cabin image. The panoramic cabin image can be used to indicate the entirety of the vehicle's cabin. Based on processing of the panoramic cabin image, such as target detection, the distribution location of objects within the vehicle's cabin can be identified. Of course, in practical applications, other methods may be combined or employed to identify the distribution location of objects, such as those based on infrared sensors or radar.

[0095] In some embodiments, step S020 may include obtaining at least one set of object status data that matches the object distribution location from the multiple sets of object status data. Each set of object status data in the multiple sets of object status data corresponds to at least one seat position in the vehicle. For example, each set of object status data may correspond to one seat position in the vehicle, or each set of object status data may correspond to two or more seat positions in the vehicle. In actual applications, the number of seat positions corresponding to each set of object status data is related to the configuration of the in-vehicle acquisition module.

[0096] Taking the example of each set of object status data corresponding to a seat position in the vehicle, embodiments of the present application can be equipped with a corresponding in-vehicle acquisition module, such as a camera or sensor, for each seat position, thereby enabling real-time monitoring of all seat positions where passengers may be present. The in-vehicle acquisition module at each seat position can generate a set of object status data. However, since there may not be passengers at one or more seat positions, the object status data acquired at these seat positions is invalid. To prevent these invalid object status data from occupying the computing power and memory bandwidth of the full-cabin service, these invalid object status data can be removed from the multiple sets of object data. That is, at least one set of object status data that matches the object distribution location is acquired from the multiple sets of object data, so that the vehicle can be further controlled to provide full-cabin service based on the acquired at least one set of object status data.

[0097] In some embodiments, the above step S100 may include the following steps:

[0098] Step S110: inputting each set of object state data into the first model so that the first model outputs object perception data;

[0099] Step S120: Controlling the vehicle to provide full-cabin service according to the object perception data corresponding to at least one set of object status data.

[0100] For each set of object status data that matches the object distribution position, the set of object status data is input into the first model so that the first model outputs object perception data; then, full-cabin service is provided based on the object perception data corresponding to at least one set of object status data that matches the object distribution position.

[0101] The embodiment of the present application does not limit the specific type of the first model, and in actual applications, it can be flexibly set in combination with the type and needs of the object state data. For example, in the case where the object state data includes an image of the object riding, the first model can be a visual perception model such as ViT (Vision Transformer). The first model can be a multimodal artificial intelligence model to adapt to the multimodal object state data. Based on the difference between the first model and the object state data, the object perception data is also different. For example, in the case where the object state data includes an image of the object riding and the first model includes a visual perception model, the object perception data includes but is not limited to the movements, behaviors, expressions, emotions, physical conditions, fatigue levels, etc. of the driving object. The embodiment of the present application realizes the automated processing of the object state data based on the first model, and accurately and efficiently obtains the object perception data.

[0102] In some embodiments, the above step S110 may include: inputting each set of object state data into the embedding module of the first model so that the embedding module outputs object feature data; inputting the object feature data corresponding to each set of object state data into the filling module of the first model so that the filling module outputs starting prediction data; inputting the starting prediction data corresponding to each set of object state data into the decoding module of the first model so that the decoding module outputs object perception data.

[0103] Exemplarily, the first model is the visual perception model ViT, and the first model may include an embedding module (Embedding), a filling module (Prefill) and a decoding module (Decode). Based on the embedding module, the object state data can be processed into object feature data, and the object feature data can be a Token vector, so as to facilitate subsequent processing of the first model. Based on the filling module, the object feature data can be processed into starting prediction data, and the starting prediction data can be a first-word Token output. Based on the decoding module, the starting prediction data can be processed into object perception data, and the object perception data can be a full reasoning Token output. Among them, the first-word Token output refers to the first Token output by the first model during the processing process, which represents the most likely starting word or character predicted by the first model based on the input; the full reasoning Token output refers to all Tokens output by the first model after completing the entire reasoning process, which can form a complete text description.

[0104] When processing the object state data based on the first model to obtain object perception data, and further providing full-cabin services based on the object perception data, since the object state data is matched with the object distribution location, the first model avoids processing invalid object state data, reduces the input of the first model, and under the same computing power and the same memory bandwidth, the first model can achieve faster inference delay, which helps to provide faster, more effective and personalized full-cabin services.

[0105] In some embodiments, step S100 may include controlling the vehicle to provide full-cabin service based on at least one set of object status data and the vehicle's driving status data. In addition to providing full-cabin service based on the object status data, embodiments of the present application also incorporate vehicle driving status data to provide full-cabin service. By integrating multimodal and multidimensional data to provide full-cabin service, the accuracy of full-cabin service can be improved.

[0106] The driving status data is used to indicate the driving status of the vehicle. To enrich and improve the driving status data, in some embodiments, the driving status data includes at least one of the following: motion characteristic data, environmental characteristic data, navigation positioning data, and interactive control data.

[0107] Motion characteristic data includes data related to vehicle operation. In some embodiments, this motion characteristic data includes at least one of the following: vehicle body posture, vehicle speed, and operating state. Vehicle body posture includes roll, suspension raise, and suspension lower; operating state includes parked, stopped, and started. Motion characteristic data can be determined based on motion control signals acquired by a control signal acquisition module. These motion control signals include, but are not limited to, body domain control signals, power domain control signals, and energy domain control signals.

[0108] Environmental characteristic data includes environmental characteristic data outside the vehicle and / or environmental characteristic data inside the vehicle. In some embodiments, the environmental characteristic data includes at least one of the following: road data, weather data, obstacle data, pedestrian data, and temperature. Road data includes data such as road conditions and road signs; temperature includes the temperature outside the vehicle and / or the temperature inside the vehicle, or may also include the temperature of the air conditioner inside the vehicle. The environmental characteristic data can be determined based on the external environmental data acquired by the external vehicle acquisition module and / or the internal environmental data acquired by the internal vehicle acquisition module.

[0109] Navigation positioning data is used to indicate the vehicle's geographic location and map information. In some embodiments, the navigation positioning data includes at least one of the following: vehicle geographic location, navigation route, gas station data, service area data, and building data. The navigation positioning data can be obtained based on the navigation positioning module.

[0110] Interactive control data refers to data generated by the interaction between the driver and the vehicle. The driver can interact with the vehicle through voice, text, gestures, touch operations, etc. Based on this, in some embodiments, interactive control data includes at least one of the following: sound data, gesture data, and touch data. Sound data can be acquired based on the acoustic module, while gesture data and touch data can be acquired based on the in-vehicle acquisition module and / or the control signal acquisition module.

[0111] In an embodiment of the present application, a vehicle is controlled to provide full-cabin services based on at least one set of object state data and the vehicle's driving state data. Specifically, the object state data can be inferred and perceived based on a first model to obtain object perception data, and then full-cabin services can be further provided based on the object perception data. The inference and perception of the vehicle's driving state data can also be implemented by an artificial intelligence model. For example, the driving state data can be inferred and perceived based on the first model to obtain driving perception data, or the driving state data can be inferred and perceived based on other models different from the first model, such as voiceprint recognition models, natural language processing, convolutional neural networks, etc., to obtain driving perception data, and then the driving perception data and object perception data can be combined to further provide full-cabin services.

[0112] In some embodiments, step S100 may include controlling the vehicle to provide full-cabin service based on at least one set of object status data when a first condition is met. Because full-cabin service requires computing resources, to avoid wasting computing resources, embodiments of the present application set a first condition to enable full-cabin service when the first condition is met.

[0113] The embodiments of this application do not limit the specific content of the first condition; in actual applications, it can be flexibly configured based on actual needs. In some embodiments, the first condition includes at least one of the following: the vehicle is in motion and a command to enable full-cabin services has been received. The driver or passenger can trigger the command to enable full-cabin services based on voice commands, touch input, or other means. Enabling full-cabin services while the vehicle is in motion can prevent them from disturbing passengers who are parked and resting; allowing them to activate upon receiving a command to enable full-cabin services can tailor the activation of full-cabin services to the personalized needs of the driver or passenger.

[0114] In some embodiments, the vehicle control method further includes: stopping the full-cabin service when a second condition is met. In this embodiment of the application, the second condition is set to stop the full-cabin service based on the second condition, which can make the full-cabin service more intelligent and humane.

[0115] The embodiments of the present application do not limit the specific content of the second condition, and it can be flexibly set according to the actual needs in actual application. In some embodiments, the second condition includes at least one of the following: receiving a stop instruction for the full-cabin service and the vehicle being stationary. The driver and passenger can trigger the stop instruction for the full-cabin service based on voice instructions, touch input, etc.; the vehicle being stationary includes the vehicle being in a parked state or a parked state. Allowing the full-cabin service to be turned off when a stop instruction for the full-cabin service is received can make the shutdown of the full-cabin service match the personalized needs of the driver and passenger; allowing the full-cabin service to be turned off when the vehicle is stationary can avoid the full-cabin service from disturbing the driver and passenger who are parked for a rest.

[0116] In some embodiments, the above-mentioned vehicle control method also includes: when the third condition is met, controlling the data acquisition system of the vehicle to enter a standby state. The data acquisition system is used to obtain the data required for full-cabin service. The embodiment of the present application sets a third condition to trigger the data acquisition system for full-cabin service to enter a standby state based on the third condition, so as to facilitate the subsequent real-time acquisition of data required for full-cabin service. The embodiment of the present application does not limit the specific content of the third condition, and it can be flexibly set in combination with needs in actual applications. In some embodiments, the third condition includes: the domain controller of the vehicle completes startup. The domain controller of the vehicle is used to control the vehicle to provide full-cabin service. When the domain controller of the vehicle completes startup, the data acquisition system is controlled to enter a standby state, so as to flexibly cooperate with the domain controller to provide the data required for full-cabin service when the domain controller needs to provide full-cabin service, thereby improving the timeliness of full-cabin service.

[0117] Below, the vehicle control method provided in the embodiments of the present application is introduced and explained with several examples.

[0118] See also Figure 3, Figure 3 This is a flow chart of another vehicle control method provided by an embodiment of the present application. This vehicle control method can be applied to the above-mentioned vehicle control system. Figure 3 As shown, the vehicle control method may include the following steps S301 to S313.

[0119] Step S301: Determine whether the driver or passenger has passed identity verification.

[0120] When a passenger approaches the vehicle, identity verification can be performed using an NFC card, an external FaceID camera, and / or other pre-defined identification methods to ensure vehicle safety. If there are multiple passengers, identity verification can be performed for one or more of them, though this is not a limitation in this embodiment of the application. If the passenger passes identity verification, step S302 below can be executed; otherwise, step S301 is repeated.

[0121] Step S302: Control the vehicle ignition start.

[0122] When the driver or passenger passes the identity authentication, the vehicle can be automatically ignited and started through the ignition control signal of the EEA.

[0123] Step S303: Open the car door.

[0124] After the vehicle is started, the doors can be opened proactively through body control signals to guide the driver and passengers aboard. This can be done by opening only the door at the driver's seat, or by identifying the location of the driver and passengers outside the vehicle using external cameras or sensors to open the doors at the corresponding positions.

[0125] Step S304: Start the domain controller.

[0126] The domain controller may be a cockpit domain controller. The domain controller may be initialized and then started normally. Step S303 and step S304 may be performed sequentially or simultaneously.

[0127] Step S305: Initialize the data acquisition system.

[0128] After the domain controller is started, all peripherals connected to the domain controller, namely the data acquisition system, including but not limited to in-vehicle and out-of-vehicle cameras, in-vehicle and out-of-vehicle sensors, communication systems, acoustic modules, navigation and positioning systems, etc., can be initialized to put the entire AI multimodal reasoning large model fusion system into a standby state. This AI multimodal reasoning large model fusion system can refer to the aforementioned vehicle control system.

[0129] Step S306: Proactively provide personalized services to the driver.

[0130] After the driver gets in the car, the AI ​​multimodal reasoning large model fusion system recognizes the driver's expression, clothing, weight and other information through voice or the car display system, and proactively greets the driver, providing humane care and personalized services, such as proactive greetings, seat adjustment, emotion recognition, music playing, air conditioning temperature and air volume adjustment, navigation, and destination inquiry.

[0131] Step S307: Determine whether the number of the riding objects is more than one.

[0132] The AI ​​multimodal reasoning large model fusion system can identify the number of occupants through in-vehicle cameras, vehicle occupancy signals, etc. If there is more than one occupant, the process proceeds to step S308; otherwise, the process proceeds to step S311.

[0133] Step S308: Determine whether the fellow passenger is riding the current vehicle for the first time.

[0134] The co-passenger object refers to other objects in the driving object except the driver. If the co-passenger object is riding the current vehicle for the first time, the following step S309 can be continued; otherwise, the following step S310 can be continued.

[0135] Step S309: Actively conduct interactive dialogue and provide interactive services.

[0136] When a fellow passenger is riding in the current vehicle for the first time, the AI ​​multimodal reasoning large model fusion system can proactively ask the car owner or fellow passengers through voice to actively engage in interactive dialogue, and save the fellow passenger's key identity information and key features through dialogue memory, so as to provide proactive care services for the next ride.

[0137] Step S310: Proactively provide personalized services to fellow passengers.

[0138] When the fellow passenger is a familiar one and this is not their first time riding in the vehicle, the AI ​​multimodal reasoning large model fusion system can proactively adjust the seat for their riding position and provide a series of multimodal reasoning and thoughtful recommendation services such as proactive care and emotion recognition based on their past riding preferences.

[0139] Step S311: Proactively provide full cabin service in a timely manner while the vehicle is traveling.

[0140] While the vehicle is in motion, the AI ​​multimodal reasoning and large-scale model fusion system can proactively provide timely full-cabin services through multimodal data collection systems such as internal and external cameras, internal and external sensors, and navigation and positioning systems. For example, it can provide reminders of upcoming gas stations, restaurants and coffee shops, supermarkets, road and vehicle conditions, and destinations.

[0141] Step S312: After arriving at the destination, provide a get-off reminder and care service.

[0142] After arriving at the destination, the AI ​​multimodal reasoning large model fusion system will proactively remind drivers and passengers to take their belongings, such as keys, mobile phones, handbags, luggage, etc., and prepare to get off the vehicle based on navigation positioning data; it will also proactively provide emotional care, such as reminders about the road conditions at the destination, pedestrian and vehicle safety, and the owner's physical condition.

[0143] Step S313: Provide automatic parking, engine shutdown and parking monitoring services.

[0144] After the driver and passengers get off the vehicle, the AI ​​multimodal reasoning large model fusion system can proactively provide services such as automatic parking, locking the vehicle, automatic engine shutdown, and parking monitoring.

[0145] See also Figure 4 , Figure 4 This is a flow chart of another vehicle control method provided by an embodiment of the present application. This vehicle control method can be applied to the above-mentioned vehicle control system. Figure 4 As shown, the vehicle control method may include the following steps S401 to S412. It should be understood that Figure 4 An example could be Figure 3 Specific execution process of step S311 in the embodiment. During the vehicle's driving process, the AI ​​multimodal reasoning large model fusion system is in the background running state and is ready to provide full cabin service at any time.

[0146] Step S401: The navigation and positioning system obtains navigation and positioning data in real time.

[0147] The navigation and positioning system can locate the current vehicle geographic location in real time and update the surrounding map information during driving in real time, including but not limited to navigation routes, gas stations, service areas, landmark buildings and other data.

[0148] Step S402: The vehicle external collection module senses the vehicle external environment data in real time.

[0149] External cameras and / or external sensors can perceive relevant data such as road conditions, road signs, surrounding environment, weather conditions, pedestrians, obstacles, etc. in real time.

[0150] Step S403: The in-vehicle collection module senses the in-vehicle environment data in real time.

[0151] The in-car camera and / or in-car sensor can perceive the behavior, expression, physical condition, in-car environment and other relevant data of the occupants in the car in real time.

[0152] Step S404: The control signal acquisition module provides motion control signals in real time.

[0153] The control signal acquisition module can provide motion control signals in real time to indicate the current vehicle operating status, body posture, speed, air conditioning, temperature and other related data in real time.

[0154] Step S405: The acoustic module processes the sound data.

[0155] The acoustic module can obtain and identify information such as the sound field inside the car, the location of the pickup area, and the identity of the driver and passengers.

[0156] Step S406: The domain controller determines whether to provide full-cabin service.

[0157] If the AI ​​multimodal reasoning large-scale model fusion system proactively perceives and needs to provide timely full-cabin services for the driver and passengers, it proceeds to step S407; otherwise, it repeats step S406. The AI ​​multimodal reasoning large-scale model fusion system can leverage the fast computing and storage capabilities of the vehicle-side multi-core heterogeneous domain controller to perform big data analysis and ultimately provide accurate, detailed, meticulous, and personalized full-cabin services.

[0158] Step S407: The in-car camera captures a cockpit panoramic image and four images of the subject riding in the cabin.

[0159] The in-car camera can capture a panoramic cabin image and four passenger images, which can correspond to the driver's seat, the front passenger seat, the left rear seat, and the right rear seat. Step S407 can be performed in real time, facilitating more accurate recognition.

[0160] Step S408: Input the five images into the ViT model so that the embedding module of the ViT model outputs the Token vectors corresponding to the five images respectively.

[0161] After the five images acquired in step S407 are input into the ViT model, the five images are transformed by the embedding module (Embedding) of the ViT model to generate Token vectors that can be processed by the Transformer model.

[0162] Step S409: The filling module and decoding module of the ViT model are successively passed to obtain all the inference Token outputs corresponding to the five images.

[0163] The Prefill module generates the first-word token output corresponding to each of the five images, completing the filling process. The Decode module generates all the inference token outputs corresponding to each of the five images, completing the inference process.

[0164] Step S410: Control the vehicle to provide full-cabin service based on the multimodal perception data.

[0165] The multimodal perception data includes all inference token outputs obtained in step S409, as well as perception data derived from the data acquired in steps S401 to S405, such as environmental feature data, motion feature data, navigation positioning data, and interactive control data. Full-cabin services include, but are not limited to, at least one of the following: upcoming gas station reminders, navigation route reminders, fatigue driving reminders, entertainment playback, restaurant and coffee shop recommendations, emotional value provision, and driver and passenger communication.

[0166] Step S411: The driving object interacts with the vehicle.

[0167] Drivers and passengers can interact with the AI ​​multimodal reasoning large model fusion system through voice, gestures, touch and other interactive methods, and put forward more personalized and customized service requirements.

[0168] Step S412: Stop all cabin services.

[0169] Drivers and passengers can proactively issue commands to stop all cabin services; alternatively, the AI ​​multimodal reasoning large model fusion system can proactively enter background operation based on pre-set conditions. For example, the AI ​​multimodal reasoning large model fusion system can proactively sense the status of the driver and passenger. If the driver and passenger stops to rest, the AI ​​multimodal reasoning large model fusion system can stop all cabin services to avoid disturbing the driver and passenger's rest.

[0170] It should be understood that the end-side AI multimodal reasoning large model fusion system needs to involve billions or even more parameter processing during the reasoning process. Therefore, the computing power, memory bandwidth, etc. of the domain controller on the end side require greater capabilities. However, the computing power and memory on the end side are limited by many factors such as chip technology, power consumption, computational complexity, and cost. It is difficult to meet users' demand for model applications with larger parameter amounts in a short period of time. Therefore, in order to deploy a large multimodal reasoning model with larger parameters on the end side based on the same computing power and the same memory bandwidth, the embodiment of the present application also proposes a method for optimizing the computing power and memory bandwidth of the end-side AI multimodal reasoning large model system.

[0171] Taking a large Transformer-based inference model as an example, its inference process passes through a padding module and a decoding module. The padding module primarily processes the input token vector in parallel, consuming significant computing power. The decoding module primarily outputs all tokens word by word based on the context token. Each token output requires retrieving the entire large model, which involves frequent memory accesses. Memory access speed directly determines the latency of a complete inference. Latency is a direct experience for passengers, so low inference latency is a key metric for large multimodal inference models. Considering the computing power and memory bandwidth requirements of these two modules, reducing the number of input tokens during an inference process can effectively reduce the computing power requirements of the padding module. Therefore, reducing the effective area of ​​the full-cabin service scenario is equivalent to reducing the token output during inference, which effectively reduces the memory bandwidth of the decoding module.

[0172] exist Figure 4 In this embodiment, the large, on-device AI multimodal inference model requires five simultaneous input images: one panoramic cabin image and four images of passengers. In a real-world cabin environment, various scenarios may exist, including one, two, three, or four passengers. Therefore, in scenarios where fewer than four passengers are in the cabin, full-cabin service push is performed only for the cabin locations where passengers are present, effectively reducing the memory bandwidth requirements of the large AI multimodal inference model.

[0173] See also Figure 5 , Figure 5 This is a flow chart of another vehicle control method provided by an embodiment of the present application. This vehicle control method can be applied to the above-mentioned vehicle control system. Figure 5 As shown, the vehicle control method may include the following steps S501 to S514. It should be understood that Figure 5 An example could be Figure 3 Specific execution process of step S311 in the embodiment. During the vehicle's driving process, the AI ​​multimodal reasoning large model fusion system is in the background running state and is ready to provide full cabin service at any time.

[0174] Step S501: The navigation and positioning system obtains navigation and positioning data in real time.

[0175] Step S502: The vehicle external collection module senses the vehicle external environment data in real time.

[0176] Step S503: The in-vehicle collection module senses the in-vehicle environment data in real time.

[0177] Step S504: The control signal acquisition module provides motion control signals in real time.

[0178] Step S505: The acoustic module processes the sound data.

[0179] Step S506: The domain controller determines whether to provide full-cabin service.

[0180] Step S507: The in-car camera captures a cockpit panoramic image and four images of the subject riding in the cabin.

[0181] For the description of steps S501 to S507, please refer to the above embodiment, such as the description of steps S401 to S407, which will not be repeated here.

[0182] Step S508: Identify the object position distribution of the passengers in the vehicle based on the cockpit panoramic image.

[0183] According to the cockpit panoramic image, the number and position of the driving objects in the vehicle are identified, and the corresponding object riding image is selected for subsequent processing based on the identification result.

[0184] Step S509: acquiring a cockpit panoramic image and at least one object riding image in real time according to the object position distribution.

[0185] For example, when it is recognized that there is a passenger at only the main driving position of the cockpit, a cockpit panoramic image and an image of the passenger at the main driving position of the cockpit are acquired in real time.

[0186] Exemplarily, when it is recognized that there are passengers at the main driver's seat and the co-pilot seat in the cabin, a panoramic image of the cabin, an image of the passengers at the main driver's seat in the cabin, and an image of the passengers at the co-pilot seat in the cabin are acquired in real time.

[0187] For example, when it is identified that there are passengers at the main driver's seat, the front passenger seat and the left rear row of the cabin, a panoramic image of the cabin, an image of the passengers at the main driver's seat, an image of the passengers at the front passenger seat and an image of the passengers at the left rear row of the cabin are acquired in real time.

[0188] For example, when it is identified that there are passengers at the main driver's seat, the front passenger seat and the right rear seat of the cabin, a panoramic image of the cabin, an image of the passengers at the main driver's seat, an image of the passengers at the front passenger seat and an image of the passengers at the right rear seat of the cabin are acquired in real time.

[0189] Exemplarily, when it is identified that there are passengers at the main driver's seat, the co-pilot's seat, the left rear row of the cabin, and the right rear row of the cabin, a panoramic image of the cabin, an image of the passengers at the main driver's seat, an image of the passengers at the co-pilot's seat, an image of the passengers at the left rear row of the cabin, and an image of the passengers at the right rear row of the cabin are acquired in real time.

[0190] Step S510: Input the cockpit panoramic image and at least one object riding image into the ViT model, so that the embedding module of the ViT model outputs a token vector corresponding to each image.

[0191] After the multiple images acquired in step S509 are input into the ViT model, the multiple images are respectively transformed by the embedding module (Embedding) of the ViT model to generate Token vectors that can be processed by the Transformer model.

[0192] Step S511: The filling module and decoding module of the ViT model are successively passed to obtain all the inference Token outputs corresponding to multiple images.

[0193] The Prefill module generates the first-word token output corresponding to multiple images, completing the filling process. The Decode module generates all the inference token outputs corresponding to multiple images, completing the inference process.

[0194] Step S512: Control the vehicle to provide full-cabin service based on the multimodal perception data.

[0195] The multimodal perception data includes all inference token outputs obtained in step S511, as well as perception data derived from the data acquired in steps S501 to S505, such as environmental feature data, motion feature data, navigation positioning data, and interactive control data. Full-cabin services include, but are not limited to, at least one of the following: upcoming gas station reminders, navigation route reminders, fatigue driving reminders, entertainment playback, restaurant and coffee shop recommendations, emotional value provision, and driver and passenger communication.

[0196] Step S513: The driving object interacts with the vehicle.

[0197] Step S514: Stop all cabin services.

[0198] For the description of steps S513 to S514, please refer to the above embodiment, as well as the description of steps S411 to S412, which will not be repeated here.

[0199] According to a third aspect of the present application, an embodiment of the present application provides a computer-readable storage medium.

[0200] The computer-readable storage medium stores a computer program or instruction. When the computer program or instruction is executed by the processor, the vehicle control method as described above is implemented and has all the beneficial effects of the vehicle control method. This application will not go into details here.

[0201] According to the fourth aspect of the present application, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the vehicle control method as described above and has all the beneficial effects of the vehicle control method. This application will not go into details here.

[0202] According to a fifth aspect of the present application, an embodiment of the present application further provides an electronic device comprising: a memory and a processor, wherein the memory stores a computer program or instructions; the processor is configured to execute the computer program or instructions in the memory to implement the steps of the above-described vehicle control method. This electronic device has all the beneficial effects of the above-described vehicle control method, and this application will not further elaborate on them.

[0203] The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof, and this application does not specifically limit this. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0204] In some embodiments of the present application, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0205] Computer program code for performing the operations of some embodiments of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider to connect via the Internet).

[0206] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function.

[0207] It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures.

[0208] For example, two blocks shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow charts, and combinations of blocks in the block diagrams and / or flow charts, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or may be implemented using a combination of dedicated hardware and computer instructions.

[0209] The units described in some embodiments of the present application may be implemented in software or hardware, and may also be provided in a processor.

[0210] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and the like.

[0211] According to the sixth aspect of this application, Figure 6 As shown, the embodiment of the present application further provides a vehicle 10, which includes the above-mentioned vehicle control system or the above-mentioned electronic device. The vehicle has all the beneficial effects of the above-mentioned vehicle control system or electronic device, which will not be described in detail in this application.

[0212] The vehicle may be a fuel vehicle, a plug-in hybrid vehicle or a new energy vehicle, etc., and this application does not make any specific restrictions on this.

[0213] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0214] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0215] The embodiments, implementation methods and related technical features of the present application can be combined and replaced with each other without conflict.

[0216] The above are merely preferred embodiments of the present application and do not constitute any form of limitation to the present application. Although the descriptions of each embodiment in the embodiments of the present application have different focuses, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A vehicle control method, characterized in that: The method comprises: controlling the vehicle to provide full cabin service based on at least one set of object status data; Wherein, at least one set of the object state data matches the object distribution position of the driving object in the vehicle.

2. The vehicle control method according to claim 1, characterized in that: The subject status data includes at least one of the following: a subject riding image, a subject voice data, a subject text data, a subject body temperature data, a subject pulse data, and a subject breathing rate.

3. The vehicle control method according to claim 1, characterized in that: The method further comprises: Identify the object distribution location of the driving objects in the vehicle; At least one set of object status data is acquired according to the object distribution positions.

4. The vehicle control method according to claim 3, characterized in that: The identifying the object distribution position of the passenger in the vehicle includes: According to the panoramic image of the vehicle cabin, object distribution positions of passengers in the vehicle are identified.

5. The vehicle control method according to claim 3, characterized in that: The acquiring at least one set of object status data according to the object distribution positions includes: Acquire at least one set of object state data that matches the object distribution position from the multiple sets of object state data; Each set of the object status data in the plurality of sets of the object status data corresponds to at least one seat position of the vehicle.

6. The vehicle control method according to claim 1, characterized in that: The controlling the vehicle to provide full-cabin service according to at least one set of object status data includes: inputting each set of the object state data into a first model so that the first model outputs object perception data; The vehicle is controlled to provide full-cabin service according to the object perception data respectively corresponding to at least one set of the object status data.

7. The vehicle control method according to claim 6, characterized in that: Inputting each set of the object state data into a first model so that the first model outputs object perception data comprises: inputting each set of the object state data into an embedding module of the first model so that the embedding module outputs object feature data; Inputting the object feature data corresponding to each set of the object state data into a filling module of the first model, so that the filling module outputs initial prediction data; The initial prediction data corresponding to each set of the object state data is input into the decoding module of the first model, so that the decoding module outputs object perception data.

8. The vehicle control method according to claim 1, wherein: The controlling the vehicle to provide full-cabin service according to at least one set of object status data includes: The vehicle is controlled to provide full-cabin service according to at least one set of object status data and the driving status data of the vehicle.

9. The vehicle control method according to claim 8, characterized in that: The driving status data includes at least one of the following: motion characteristic data, environmental characteristic data, navigation positioning data, and interactive control data.

10. The vehicle control method according to claim 9, characterized in that: The motion characteristic data includes at least one of the following: vehicle body posture, vehicle speed, and operating status.

11. The vehicle control method according to claim 9, characterized in that: The environmental characteristic data includes at least one of the following: road data, weather data, obstacle data, pedestrian data, and temperature.

12. The vehicle control method according to claim 9, characterized in that: The navigation positioning data includes at least one of the following: vehicle geographic location, navigation route, gas station data, service area data, and building data.

13. The vehicle control method according to claim 9, characterized in that: The interactive control data includes at least one of the following: sound data, gesture data, and touch data.

14. The vehicle control method according to claim 1, characterized in that: The controlling the vehicle to provide full-cabin service according to at least one set of object status data includes: When a first condition is met, the vehicle is controlled to provide full-cabin service according to at least one set of object status data.

15. The vehicle control method according to claim 14, characterized in that: The first condition includes at least one of the following: the vehicle is in motion and an enabling instruction for the full-cabin service is received.

16. The vehicle control method according to claim 1, characterized in that: The method further comprises: When the second condition is met, the full-cabin service is stopped.

17. The vehicle control method according to claim 16, characterized in that: The second condition includes at least one of the following: receiving a stop instruction for the full-cabin service, and the vehicle being stationary.

18. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the vehicle control method according to any one of claims 1 to 17 is implemented.

19. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the vehicle control method according to any one of claims 1 to 17 is implemented.

20. An electronic device, characterized in that: include: Memory on which computer programs or instructions are stored; A processor, configured to execute the computer program or instructions in the memory to implement the vehicle control method according to any one of claims 1 to 17.

21. A vehicle control system, characterized in that: The vehicle control system includes: A domain controller (100) is used to control a vehicle to provide a full-cabin service based on at least one set of object status data; wherein at least one set of the object status data matches the object distribution position of objects driven in the vehicle.

22. The vehicle control system according to claim 21, characterized in that: The domain controller (100) includes at least one of the following: a cockpit domain controller and a cabin-cockpit integrated central computing domain controller.

23. The vehicle control system according to claim 21, characterized in that: The vehicle control system further includes: A data acquisition system (200) is connected to the domain controller (100) and is used to obtain data required for the full cabin service.

24. The vehicle control system according to claim 23, characterized in that: The data acquisition system (200) comprises: A control signal acquisition module (210) for acquiring a motion control signal of the vehicle; The motion control signal is used to determine motion characteristic data of the vehicle.

25. The vehicle control system according to claim 24, characterized in that: The motion control signal includes at least one of the following: a body domain control signal, a power domain control signal, and an energy domain control signal.

26. The vehicle control system according to claim 23, characterized in that The data acquisition system (200) comprises: An external vehicle acquisition module (220) for acquiring external environment data of the vehicle; The external environment data is used to determine the environmental characteristic data of the vehicle.

27. The vehicle control system according to claim 26, characterized in that: The off-vehicle acquisition module (220) includes at least one of the following: an off-vehicle camera unit (221) and an off-vehicle sensor unit (222).

28. The vehicle control system according to claim 27, characterized in that: The vehicle exterior camera unit (221) includes at least one of the following: a vehicle front camera, a vehicle rear camera, a vehicle left camera, a vehicle right camera, and a face recognition camera.

29. The vehicle control system according to claim 27, wherein: The off-vehicle sensor unit (222) includes at least one of the following: a temperature sensor, a humidity sensor, and an air pressure sensor.

30. The vehicle control system according to claim 23, wherein: The data acquisition system (200) comprises: An in-vehicle acquisition module (230) for acquiring in-vehicle environment data of the vehicle; The in-vehicle environment data is used to determine at least one of the object state data, the object distribution position of the driving object, and the environmental characteristic data of the vehicle.

31. The vehicle control system according to claim 30, characterized in that: The in-vehicle acquisition module (230) includes at least one of the following: an in-vehicle camera unit (231) and an in-vehicle sensor unit (232).

32. The vehicle control system according to claim 31, characterized in that: The in-vehicle camera unit (231) includes at least one of the following: a DMS, an OMS, an IMS, and an iris recognizer.

33. The vehicle control system according to claim 31, characterized in that The in-vehicle sensor unit (232) includes at least one of the following: an in-vehicle radar, an IMU, and an occupancy detector.

34. The vehicle control system according to claim 23, wherein: The data acquisition system (200) comprises: The navigation and positioning module (240) is used to obtain navigation and positioning data of the vehicle.

35. The vehicle control system according to claim 34, characterized in that The navigation and positioning module (240) includes at least one of the following: GPS, Beidou, wireless base station positioning unit, and IMU.

36. The vehicle control system according to claim 23, wherein: The data acquisition system (200) comprises: an acoustic module (250) for processing sound data; The sound data is used to determine the interactive control data of the vehicle or to provide the full-cabin service.

37. The vehicle control system according to claim 36, characterized in that: The acoustic module (250) includes at least one of the following: a microphone, a speaker, an amplifier, a vibration motor, and A2B.

38. The vehicle control system according to claim 23, wherein: The vehicle control system further includes: A communication system (300) is connected to the domain controller (100) and the data acquisition system (200), and is used to transmit data required for the full cabin service between the domain controller (100) and the data acquisition system (200).

39. A vehicle, characterized in that: The vehicle includes the electronic device according to claim 20, or includes the vehicle control system according to any one of claims 21 to 38.

Images