Control method and apparatus, and vehicle

Abstract

The present application provides a control method and apparatus, and a vehicle. The method comprises: acquiring first data acquired by a sensor in a cabin; and upon determining, on the basis of the first data, that a first user intends to sing, activating a singing mode. Activating the singing mode comprises: on the basis of a first region selected by the first user, activating a singing mode for the first region, or activating a singing mode for all regions of the cabin. By means of the above method, it is possible to automatically detect whether a user inside a vehicle intends to sing, and, upon detecting that the user inside the vehicle intends to sing, a singing mode can be activated. In this way, spontaneous singing needs of the user inside the vehicle can be quickly satisfied, thereby improving the human-computer interaction experience of the user.

Description

Control methods, devices and vehicles

[0001] This application claims priority to Chinese patent application filed on August 30, 2024, with application number 202411219264.6 and title "Control Method, Apparatus and Vehicle", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of vehicles, and more specifically, to a control method, apparatus, and vehicle. Background Technology

[0003] As smart cars become more widely used in daily life, users expect them and their related devices to provide a more comfortable and intelligent experience. Against this backdrop, in-car karaoke modes are gaining increasing popularity among users.

[0004] In most scenarios, the karaoke mode is off by default so as not to interfere with passengers' normal conversations and rest. If passengers want to sing along or duet while music is playing, they need to manually tap the display screen to turn on the karaoke mode, which will affect their entertainment experience. Summary of the Invention

[0005] This application provides a control method, device, and vehicle that can automatically detect whether a user in the vehicle intends to sing. When the intention to sing is detected, the vehicle can control the singing mode to be turned on, thereby quickly meeting the user's impromptu singing needs and improving the user's human-computer interaction experience.

[0006] In a first aspect, a control method is provided, the method comprising: acquiring first data collected by sensors in the cockpit; and activating a singing mode when it is determined from the first data that a first user intends to sing; wherein activating the singing mode comprises: activating a singing mode for a first area selected by the first user, or activating a singing mode for all areas of the cockpit.

[0007] The aforementioned "singing mode" can be understood as follows: passengers in the car can use the in-car microphone instead of an external microphone for voice input, and the car's audio system will output the user's singing voice. For example, if the driver activates the car's singing mode, the driver can sing through the in-car microphone from the driver's seat, and the car's audio system will play the driver's singing along with background music.

[0008] In one possible implementation, the vehicle can be divided into different zones (including a first zone). Users in the first zone can use the vehicle's onboard microphone (corresponding to that zone) instead of an external microphone for voice input, and the corresponding speaker in that zone can output their singing voice. For example, the vehicle can be divided into four zones: zone 1, zone 2, zone 3, and zone 4. Zone 1 can correspond to the driver's seat, zone 2 to the passenger seat, zone 3 to the left rear seat, and zone 4 to the right rear seat. The first zone can be any one of these four zones. Alternatively, the vehicle can be divided into three zones: zone 1, zone 2, and zone 3. Zone 1 can correspond to the driver's seat, zone 2 to the passenger seat, and zone 3 to the entire rear seat. The first zone can be any one of these three zones.

[0009] In this embodiment, the system can automatically detect whether a user in the vehicle intends to sing. Upon detection, it activates a singing mode, quickly satisfying the user's spontaneous singing needs and improving the human-computer interaction experience. Furthermore, the control method can also activate different singing modes according to the user's needs, enhancing the flexibility and personalization of in-vehicle entertainment.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: a control prompting device prompting the activation of the singing mode.

[0011] In this embodiment, the vehicle can control a prompting device to prompt the user to turn on the singing mode. This helps ensure that the user is aware of when the singing mode is turned on, and at the same time, it can further enhance the human-computer interaction experience of the in-vehicle entertainment system.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: when a second user in the cabin has a do-not-disturb requirement, keeping the singing mode in the second area of ​​the cabin off or off, wherein the second user is located in the second area.

[0013] In one possible implementation, detecting a second user's need to avoid disturbance includes detecting that the second user is in a state of rest, watching videos, or playing games.

[0014] In this embodiment, when the vehicle detects that the second user has a do-not-disturb requirement, the vehicle can keep the singing mode of the second area where the second user is located off or off. In this way, the vehicle microphone corresponding to the second area will not collect the voice of the second user, and the audio system corresponding to the second area will not play the singing voice of the first user. In this way, the singing needs of different users in the car can be better taken into account, thereby improving the entertainment experience of different users in the car.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, activating the singing mode includes: activating the singing mode of the first area, and the method further includes: when it is determined from the second data that a third user in the cabin intends to sing, activating the singing mode of all areas in the cabin or activating the singing mode of the third area in the cabin, wherein the third user is located in the third area.

[0016] In this embodiment, when the singing mode of the first area is activated, if the vehicle detects that a third user in the vehicle also intends to sing, the vehicle can activate the singing mode of all areas in the cabin or activate the singing mode of the third area where the third user is located. In this way, the singing needs of all users in the vehicle can be better taken into account, thereby providing a good singing environment for all users in the vehicle.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the first data includes: image information of the first user and first voice information emitted by the first user, wherein the image information of the first user includes first lip image information, and the method further includes: determining a first song based on the first lip image information and the first voice information; and controlling the sound-emitting device to play the first song.

[0018] In this embodiment, when no music is playing in the vehicle, the system can determine the first user's intention to sing based on lip reading and speech recognition results, and then play the first song hummed by the first user. In this way, the user can trigger song playback with simple lip reading and speech, without complex operations or device interaction, ensuring that the user's singing needs are met instantly.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the image information of the first user further includes: first body language information, and determining that the first user has the intention to sing based on the first data includes: determining that the first user has the intention to sing when the first user makes a first action based on the first body language information, or when the first lip shape is identified based on the first lip image information.

[0020] In one possible implementation, the first body language information may include, but is not limited to, the first user's head movements, hand gestures, and facial expression changes, etc.

[0021] In one possible implementation, recognizing the first user's first action based on the first body language information may include: recognizing the first user nodding, or recognizing the first user clapping or snapping their fingers.

[0022] In this embodiment, the intention of the first user to sing can be identified based on the first user's body language information or the first lip image information. In this way, the identification of the user's intention to sing can be more accurate.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: if no second voice information emitted by the first user is detected within a first preset time period, or if the second voice information is detected within the first preset time period and does not match the text content or music waveform corresponding to the first song, the singing mode is turned off.

[0024] In this embodiment, after detecting that the user has no intention to sing, the vehicle can control the singing mode to turn off. This automated management not only makes the in-vehicle entertainment system more intelligent, convenient and efficient, but also helps to save energy and protect the environment.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the first data includes: third voice information emitted by the first user; before activating the singing mode, the method further includes: controlling the sound-emitting device to play a second song; determining that the first user has the intention to sing based on the first data includes: determining that the first user has the intention to sing when the text content corresponding to the third voice information matches the text content corresponding to the second song.

[0026] In this embodiment of the application, when a second song is played in the vehicle, the vehicle can determine that the first user has the intention to sing based on the third voice information issued by the first user and the text content of the second song. In this way, the singing intention of the first user can be accurately and efficiently identified, thereby enabling the singing function.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, before determining that the first user has the intention to sing, the method further includes: determining that the waveform of the third voice information matches the music waveform of the second song.

[0028] In this embodiment of the application, when a second song is played in the vehicle, the vehicle can also use the waveform of the third voice information and the waveform of the second song to help identify whether the first user intends to sing. In this way, the intention of the first user to sing can be identified more accurately, thereby meeting the impromptu singing needs of the users in the vehicle.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, the first data includes: image information of the first user, the image information of the first user includes: second lip image information, and before the singing mode is activated, the method further includes: controlling the sound-generating device to play a second song; determining that the first user has the intention to sing based on the first data includes: determining that the first user has the intention to sing when the text content identified based on the second lip image information matches the text content corresponding to the second song.

[0030] In this embodiment of the application, when a second song is played in the vehicle, the vehicle can also identify the first user's intention to sing based on the second lip image information and the text content corresponding to the second song. In this way, the first user's intention to sing can be identified accurately and efficiently, thereby enabling the singing function.

[0031] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: if no fourth voice information emitted by the first user is detected within a second preset time period, or if the fourth voice information is detected to be mismatched with the text content or music waveform corresponding to the second song within the second preset time period, the singing mode is turned off.

[0032] In this embodiment, after detecting that the user has no intention to sing, the vehicle can control the singing mode to turn off. This automated management makes the in-vehicle entertainment system more intelligent, convenient and efficient, and also helps to save energy and protect the environment.

[0033] Secondly, a control method is provided, the method comprising: acquiring first data collected by sensors in the cockpit, the first data including: image information of a first user, the image information of the first user including: first lip image information and first body language information; determining that the first user intends to sing when the first user makes a first action based on the first body language information, or when the first lip shape is identified based on the first lip image information; and activating a singing mode.

[0034] In this embodiment, the system can identify the first user's intention to sing based on the first user's body language information or lip image information, and control the singing mode to be turned on. This can quickly meet the impromptu singing needs of users in the car and improve the user's human-computer interaction experience.

[0035] In conjunction with the second aspect, in some implementations of the second aspect, the first data further includes first voice information emitted by the first user, and the method further includes: determining a first song based on the first lip image information and the first voice information; and controlling the sound-emitting device to play the first song.

[0036] In this embodiment, when no music is playing in the vehicle, the system can play a first song hummed by the first user based on the lip-reading and speech recognition results. In this way, the user can trigger song playback with simple lip-reading and speech, without complex operations or device interaction, ensuring that the user's singing needs are met instantly.

[0037] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes: if no second voice information emitted by the first user is detected within a first preset duration, or if the second voice information is detected within the first preset duration and does not match the text content or music waveform corresponding to the first song, the singing mode is turned off.

[0038] In this embodiment, after detecting that the user has no intention to sing, the vehicle can control the singing mode to turn off. This automated management makes the in-vehicle entertainment system more intelligent, convenient and efficient, and also helps to save energy and protect the environment.

[0039] In conjunction with the second aspect, in some implementations of the second aspect, the first data further includes third voice information emitted by the first user; before acquiring the first data collected by the sensors in the cockpit, the method further includes: controlling the sound-emitting device to play a second song; before determining that the first user intends to sing, the method further includes: determining that the third voice information matches the text content or music waveform corresponding to the second song.

[0040] In this embodiment of the application, when music is playing in the vehicle, the vehicle can determine that the first user has the intention to sing based on the third voice information issued by the first user and the text content or music waveform of the second song. In this way, the singing intention of the first user can be accurately and efficiently identified, thereby activating the singing function.

[0041] In conjunction with the second aspect, in some implementations of the second aspect, before determining that the first user has the intention to sing, the method further includes: determining that the text content corresponding to the first lip image information matches the text content corresponding to the second song.

[0042] In this embodiment of the application, when music is playing in the car, the vehicle can identify the first user's intention to sing based on the first lip image information and the text content corresponding to the second song. In this way, the first user's intention to sing can be identified accurately and efficiently, thereby enabling the singing function.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, if the fourth voice information emitted by the first user is not detected within the second preset time period, or if the fourth voice information is detected to be mismatched with the text content or music waveform corresponding to the second song within the second preset time period, the singing mode is turned off.

[0044] In this embodiment, after detecting that the user has no intention to sing, the vehicle can control the singing mode to turn off. This automated management not only makes the in-vehicle entertainment system more intelligent, convenient and efficient, but also helps to save energy and protect the environment.

[0045] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes: a control prompting device prompting the user to activate the singing mode.

[0046] In this embodiment, before the singing mode is activated, the vehicle can control a prompting device to indicate that the singing mode is activated. This helps to ensure that the user is aware of the activation of the singing mode and further enhances the human-computer interaction experience of the in-vehicle entertainment system.

[0047] In conjunction with the second aspect, in some implementations of the second aspect, activating the singing mode includes: activating the singing mode of the first area selected by the first user, or activating the singing mode of all areas of the cabin.

[0048] In this embodiment, different singing modes can be enabled according to the user's needs, which can improve the flexibility and personalization of in-vehicle entertainment.

[0049] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes: when the second user in the cabin has a do-not-disturb requirement, keeping the singing mode in the second area of ​​the cabin off or turning it off, with the second user located in the second area.

[0050] In this embodiment, when the vehicle detects that the second user has a do-not-disturb requirement, the vehicle can turn off the singing mode in the second area where the second user is located. In this way, the vehicle microphone corresponding to the second area will not collect the voice of the second user, and the audio system corresponding to the second area will not play the singing of the first user. In this way, the singing needs of different users in the car can be better taken into account, thereby improving the entertainment experience of different users in the car.

[0051] In conjunction with the second aspect, in some implementations of the second aspect, activating the singing mode includes: activating the singing mode of the first area, and the method further includes: when it is determined from the second data that a third user in the cabin intends to sing, activating the singing mode of all areas of the cabin or activating the singing mode of the third area in the cabin, wherein the third user is located in the third area.

[0052] In this embodiment, after detecting that the user has no intention to sing, the vehicle can control the singing mode to turn off. This automated management not only makes the in-vehicle entertainment system more intelligent, convenient and efficient, but also helps to save energy and protect the environment.

[0053] Thirdly, a control device is provided, the device comprising an acquisition unit and a processing unit; the acquisition unit is used to acquire first data collected by sensors in the cockpit; the processing unit is used to activate a singing mode when it is determined from the first data that a first user intends to sing; specifically, the processing unit is used to activate a singing mode for a first area selected by the first user, or a singing mode for all areas of the cockpit.

[0054] In conjunction with the third aspect, in some implementations of the third aspect, the processing unit is further configured to control the prompting device to prompt the activation of the singing mode.

[0055] In conjunction with the third aspect, in some implementations of the third aspect, the processing unit is further configured to keep the singing mode of the second area in the cabin off or off when the second user in the cabin has a do-not-disturb requirement, wherein the second user is located in the second area.

[0056] In conjunction with the third aspect, in some implementations of the third aspect, the processing unit is specifically used to enable the singing mode of the first area; the processing unit is also used to enable the singing mode of all areas of the cabin or enable the singing mode of the third area of ​​the cabin when it is determined from the second data that the third user in the cabin intends to sing, wherein the third user is located in the third area.

[0057] In conjunction with the third aspect, in some implementations of the third aspect, the first data includes: image information of the first user and first voice information emitted by the first user, wherein the image information of the first user includes first lip image information, and the processing unit is specifically used to: determine a first song based on the first lip image information and the first voice information; and control the sound-emitting device to play the first song.

[0058] In conjunction with the third aspect, in some implementations of the third aspect, the image information of the first user further includes: first body language information. The processing unit is specifically used to determine that the first user has the intention to sing when the first user makes a first action based on the first body language information, or when the first lip shape is identified based on the first lip image information.

[0059] In conjunction with the third aspect, in some implementations of the third aspect, the processing unit is further configured to close the singing mode if the second voice information emitted by the first user is not detected within a first preset time period, or if the second voice information is detected within the first preset time period and does not match the text content or music waveform corresponding to the first song.

[0060] In conjunction with the third aspect, in some implementations of the third aspect, the first data includes: third voice information emitted by the first user; the processing unit is further configured to control the sound-emitting device to play a second song; specifically, the processing unit is configured to determine that the first user has the intention to sing when the text content corresponding to the third voice information matches the text content corresponding to the second song.

[0061] In conjunction with the third aspect, in some implementations of the third aspect, the processing unit is further configured to determine that the waveform of the third voice information matches the music waveform of the second song.

[0062] In conjunction with the third aspect, in some implementations of the third aspect, the first data includes: image information of the first user, the image information of the first user includes: second lip image information, and the processing unit is further configured to control the sound-generating device to play a second song; the processing unit is specifically configured to determine that the first user has the intention to sing when the second text content identified based on the second lip image information matches the text content corresponding to the second song.

[0063] In conjunction with the third aspect, in some implementations of the third aspect, the processing unit is further configured to close the singing mode if the fourth voice information emitted by the first user is not detected within the second preset time period, or if the fourth voice information is detected to be mismatched with the text content or music waveform corresponding to the second song within the second preset time period.

[0064] Fourthly, a control device is provided, comprising an acquisition unit and a processing unit; the acquisition unit is used to acquire first data collected by sensors in the cockpit, the first data including: image information of a first user, the image information of the first user including: first lip image information and first body language information; the processing unit is used to: determine that the first user has the intention to sing when the first user makes a first action based on the first body language information, or when the first lip shape is identified based on the third lip image information; and activate the singing mode.

[0065] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the first data further includes first voice information emitted by the first user, and the processing unit is further configured to: determine a first song based on the first lip image information and the first voice information; and control the sound-emitting device to play the first song.

[0066] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processing unit is further configured to close the singing mode if the second voice information emitted by the first user is not detected within a first preset time period, or if the second voice information is detected within the first preset time period and does not match the text content or music waveform corresponding to the first song.

[0067] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the first data further includes third voice information emitted by the first user, and the processing unit is further configured to: control the sound-emitting device to play a second song; and determine that the third voice information matches the text content or music waveform corresponding to the second song.

[0068] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processing unit is further configured to determine that the text content corresponding to the first lip image information matches the text content corresponding to the second song.

[0069] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processing unit is further configured to close the singing mode if the fourth voice information emitted by the first user is not detected within the second preset time period, or if the fourth voice information is detected to be mismatched with the text content or music waveform corresponding to the second song within the second preset time period.

[0070] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processing unit is further configured to control the prompting device to prompt the activation of the singing mode.

[0071] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processing unit is specifically used to enable the singing mode of the first area selected by the first user, or to enable the singing mode of all areas of the cabin.

[0072] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processing unit is specifically used to keep the singing mode in the second area of ​​the cabin off or turn it off when the second user in the cabin has a do-not-disturb requirement, and the second user is located in the second area.

[0073] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processing unit is specifically used to enable the singing mode of the first area. The processing unit is also used to enable the singing mode of all areas of the cabin or enable the singing mode of the third area of ​​the cabin when it is determined from the second data that the third user in the cabin intends to sing. The third user is located in the third area.

[0074] Fifthly, a control device is provided, comprising: at least one processor and a memory, wherein the at least one processor is coupled to the memory for reading and executing instructions in the memory, such that the device implements the method in any one of the implementations of the first or second aspect described above.

[0075] Sixthly, a computer-readable storage medium is provided, the computer-readable storage medium storing program code that, when the computer program code is run on a computer, causes the computer to perform the method in any one of the implementations of the first or second aspect described above.

[0076] In a seventh aspect, a chip is provided, the chip including circuitry for performing the method in any one of the implementations of the first or second aspect described above.

[0077] Eighthly, a computer program product is provided, the computer product including a computer program that, when the computer program is run by a processor, causes the method in any of the implementations of the first or second aspect to be executed.

[0078] Ninthly, a vehicle is provided, comprising: a control device according to any one of the implementations of the third or fourth aspect described above. Attached Figure Description

[0079] Figure 1 is a functional schematic diagram of a vehicle provided in an embodiment of this application;

[0080] Figure 2 is a top view of a vehicle provided in an embodiment of this application;

[0081] Figure 3 is a schematic flowchart of a control method provided in an embodiment of this application;

[0082] Figure 4 is a schematic flowchart of another control method provided in an embodiment of this application;

[0083] Figure 5 is a schematic flowchart of another control method provided in an embodiment of this application;

[0084] Figure 6 is a schematic flowchart of another control method provided in an embodiment of this application;

[0085] Figure 7 is a schematic flowchart of another control method provided in an embodiment of this application;

[0086] Figure 8 is a schematic flowchart of another control method provided in an embodiment of this application;

[0087] Figures 9 to 15 are schematic diagrams of an application scenario to which the control method provided in the embodiments of this application is applicable;

[0088] Figures 16 to 19 are schematic diagrams of another application scenario to which the control method provided in the embodiments of this application is applicable;

[0089] Figures 20 to 22 are schematic diagrams of another application scenario to which the control method provided in the embodiments of this application is applicable;

[0090] Figure 23 is a schematic diagram of a control device provided in an embodiment of this application;

[0091] Figure 24 is a schematic diagram of another control device provided in an embodiment of this application. Detailed Implementation

[0092] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In this application, "at least one" means one or more, and "more" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0093] The use of prefixes such as "first" and "second" in this application embodiment is solely for distinguishing different descriptive objects and does not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes to distinguish descriptive objects in this application embodiment does not constitute a limitation on the described objects. The description of the described objects is found in the claims or the context of the embodiments, and the use of such prefixes should not constitute unnecessary restrictions.

[0094] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0095] Figure 1 is a functional schematic diagram of a vehicle 100 provided in an embodiment of this application.

[0096] Vehicle 100 may include multiple subsystems, such as a perception system 120, a computing platform 130, and an audio system 140. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include one or more components. Furthermore, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0097] The perception system 120 may include several types of sensors for sensing information about the environment surrounding the vehicle 100. For example, the perception system 120 may include a positioning system, which may be a global positioning system (GPS), a BeiDou system, or another positioning system. The perception system 120 may include one or more of the following: an inertial measurement unit (IMU), lidar, millimeter-wave radar, ultrasonic radar, and a camera device.

[0098] Some or all of the functions of vehicle 100 can be controlled by computing platform 130. Computing platform 130 may include processors 131 to 13n (n being a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement some or all of the functions of the aforementioned units. Furthermore, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. In addition, the computing platform 130 may also include a memory for storing instructions. Some or all of the processors 131 to 13n can call the instructions in the memory to implement the corresponding functions.

[0099] The computing platform 130 can control the functions of the vehicle 100 based on inputs received from various subsystems (e.g., the sensing system 120). In some embodiments, the computing platform 130 can be used to provide control over many aspects of the vehicle 100 and its subsystems.

[0100] The audio system 140 is capable of receiving user voice information and playing audio.

[0101] Optionally, the above components are just an example. In actual applications, the components in each of the above modules may be added or deleted as needed.

[0102] The vehicle 100 in this application may include: road vehicles, water vehicles, air vehicles, industrial equipment, agricultural equipment, or entertainment equipment, etc. For example, vehicle 100 may be a means of transportation (such as commercial vehicles, passenger cars, motorcycles, flying cars, trains, etc.), industrial vehicles (such as forklifts, trailers, tractors, etc.), engineering vehicles (such as excavators, bulldozers, cranes, etc.), agricultural equipment (such as lawnmowers, harvesters, etc.), amusement equipment, toy vehicles, etc. The embodiments of this application do not specifically limit the type of vehicle.

[0103] The following uses vehicle 100 as an example of an intelligent vehicle to illustrate the technical problems that this application needs to solve and the technical solutions adopted.

[0104] As smart cars become more widely used in daily life, users expect them and their related devices to provide a more comfortable and intelligent experience. Against this backdrop, in-car karaoke modes are gaining increasing popularity among users.

[0105] When the karaoke mode is activated, passengers can use the in-car microphone instead of an external microphone and enjoy surround sound from the car's speakers to sing karaoke anytime, anywhere. However, in most situations, to avoid interfering with passengers' conversations and rest, the karaoke mode is off by default. When passengers want to sing along or duet while listening to music, they need to manually activate the karaoke mode, which will affect their entertainment experience.

[0106] This application provides a control method, device, and vehicle that can automatically detect whether a user in the vehicle intends to sing. When the intention to sing is detected, the vehicle can control the singing mode to be turned on, thus quickly meeting the impromptu singing needs of the user and improving the user's human-computer interaction experience.

[0107] Figure 2 is a top view of a vehicle provided in an embodiment of this application.

[0108] As shown in Figure 2, the vehicle's cockpit includes: a cockpit controller 201, an in-vehicle display screen 202, an in-vehicle microphone 203, an in-vehicle speaker 204, and a camera device 205.

[0109] The cockpit controller 201 has data processing capabilities, enabling it to analyze and process data collected by sensors and control other devices based on the data processing results. This cockpit controller can be located within the computing platform 130 shown in Figure 1. The cockpit controller 201 is connected to and interacts with the in-vehicle display screen 202, in-vehicle microphone 203, in-vehicle speaker 204, and camera device 205. For example, the cockpit controller 201 can acquire a click event from a user on the in-vehicle display screen 202, perform corresponding logical processing, and then return the resulting image to the in-vehicle display screen 202 for display.

[0110] The in-vehicle display screen 202 is a display device used to display information such as images or data to users, for example, displaying signals processed by the cockpit controller 201 on the display screen.

[0111] The vehicle microphone 203 is a microphone used to receive the user's voice information, convert the sound signal into an electrical signal, and then transmit it to the cockpit controller 201 for processing.

[0112] The vehicle speaker 204 is an electroacoustic transducer that can convert the electrical signal processed by the cockpit controller 201 into an acoustic signal and finally play it in the cockpit. The vehicle microphone 203 and the vehicle speaker 204 can be located in the audio system 140 in Figure 1.

[0113] The camera device 205 is an image acquisition component that converts the acquired image data into digital signals that can be recognized by the cockpit controller 201 through a photosensitive component and a control component, and then transmits the digital signals to the cockpit controller 201 for processing.

[0114] Furthermore, the vehicle can be divided into four audio zones: Zone 1, Zone 2, Zone 3, and Zone 4. Zone 1 corresponds to the driver's area, Zone 2 to the front passenger area, Zone 3 to the left rear seat area, and Zone 4 to the right rear seat area. When the singing mode is activated in a particular zone, the user in that zone can sing. For example, when the singing mode is activated in Zone 1, the driver can sing using the microphone and speakers in the driver's area. Microphones in other zones of the cabin will not pick up the driver's or other users' voices, and speakers in other zones will not play the driver's singing.

[0115] It should be noted that the internal components of the vehicle shown in Figure 2 are merely illustrative examples, and those skilled in the art can add or remove internal components of the vehicle according to actual needs.

[0116] It should also be noted that the division of the audio zones in the vehicle shown in Figure 2 is only an illustrative example. Those skilled in the art can change the division of the audio zones according to actual needs. For example, the driver's area can be divided into audio zone 1, the passenger area into audio zone 2, and the rear area of ​​the vehicle into audio zone 3, etc.

[0117] Figure 3 is a schematic flowchart of a control method provided in an embodiment of this application. The execution subject of method 300 can be a vehicle. When the execution subject of method 300 is vehicle 100, it can be executed by computing platform 130 in vehicle 100, or by system-on-chip (SoC) in computing platform 130, or by processor in computing platform 130. The following describes method 300 with vehicle as the execution subject. Method 300 can include steps S301 to S302.

[0118] S301, acquire the first data collected by the sensors inside the cockpit.

[0119] Optionally, the first data collected by the sensor may include, but is not limited to, the user's image information and the user's voice information.

[0120] S302, when it is determined from the first data that the first user intends to sing, the singing mode is activated.

[0121] The control of the singing mode includes: enabling the singing mode of the first area selected by the first user, or enabling the singing mode of the entire cabin.

[0122] In this context, activating the singing mode in all areas of the cabin can also be understood as activating the singing mode for the entire car.

[0123] It should be noted that activating the singing mode can be understood as: the user in the car can use the in-car microphone instead of an external microphone for voice input, and the in-car audio system will output the user's singing voice. For example, if the driver activates the car's singing mode, the driver can sing through the in-car microphone from the driver's seat, and the in-car audio system will play the driver's singing along with background music. Optionally, the background music may or may not include the original vocals of the song (i.e., the corresponding karaoke mode).

[0124] It should also be noted that different zones (including the first zone) can be divided in the vehicle. Users in the first zone can use the onboard microphone corresponding to that zone instead of an external microphone for voice input, and the corresponding speaker in that zone can output the user's singing voice.

[0125] For example, in Figure 2, the vehicle can be divided into four audio zones: zone 1, zone 2, zone 3, and zone 4. Zone 1 corresponds to the driver's side, zone 2 to the front passenger side, zone 3 to the left rear seat, and zone 4 to the right rear seat. The first zone can be any one of these four zones. As another example, the vehicle can be divided into three audio zones: zone 1, zone 2, and zone 3. Zone 1 corresponds to the driver's side, zone 2 to the front passenger side, and zone 3 to the entire rear seat. The first zone can be any one of these three zones.

[0126] In this embodiment, the system can automatically detect whether a user in the vehicle intends to sing. Upon detection, it activates a singing mode, quickly satisfying the user's spontaneous singing needs and improving the human-computer interaction experience. Furthermore, the control method can also activate different singing modes according to the user's needs, enhancing the flexibility and personalization of in-vehicle entertainment.

[0127] In one possible implementation, before step S302, method 300 further includes: a control prompt device prompting the user to activate the singing mode; correspondingly, the vehicle can activate the singing mode in the first area or the singing mode for the entire vehicle based on the user's selection. This approach helps ensure the user is aware of when the singing mode is activated and further enhances the human-computer interaction experience of the in-vehicle entertainment system.

[0128] When multiple users are in the cabin, the vehicle can also take into account the singing needs of different users and turn on or off the singing mode in specific areas.

[0129] In one possible implementation, when the vehicle detects a second user in the cabin who requires a do-not-disturb setting, the vehicle can keep the karaoke mode in the second area of ​​the cabin (where the second user is located) either off or off. In other words, if the area where the karaoke mode is active overlaps with the area where the user with the do-not-disturb setting is located, then the karaoke mode will not be activated or deactivated in the area where the user requiring do-not-disturb setting resides.

[0130] For example, the first region is the region corresponding to pitch 1 in Figure 2, and the second region is the region corresponding to pitch 2 in Figure 2.

[0131] Optionally, detecting that the second user is in a state of do-not-disturbance includes detecting that the second user is in a state of rest, watching videos, or playing games.

[0132] In one possible implementation, in step S302, when the vehicle detects that a third user in the cabin intends to sing based on the second data, the vehicle can activate the singing mode for all areas of the cabin or activate the singing mode for the third area of ​​the cabin, where the third user is located in the third area. This better caters to the singing needs of all users in the vehicle, thus providing a comfortable singing environment for everyone.

[0133] For example, the first region is the region corresponding to pitch 1 in Figure 2, and the third region is the region corresponding to pitch 3 in Figure 2.

[0134] Optionally, if no music is playing in the vehicle, and the vehicle detects that the first user is singing song 1 based on the first data, and the vehicle detects that the third user is singing song 2 based on the second data, the vehicle can control the prompting device to prompt the user (the first user or the third user) to choose to sing song 1 or song 2.

[0135] In one embodiment, the first data collected by the sensor includes: image information of a first user and first voice information emitted by the first user, wherein the image information of the first user includes first lip image information. Method 300 further includes: determining a first song based on the first lip image information and the first voice information; and controlling the sound-producing device to play the first song. In this way, the user can trigger song playback with simple lip movements and voice commands, without the need for complex operations or device interaction, ensuring that the user's singing needs are met instantly.

[0136] Optionally, determining the first song based on the first lip image information and the first voice information includes determining the first song based on the text content corresponding to the first lip image information and the text content of the first voice information.

[0137] When a vehicle determines that a first user intends to sing based on the first data, it can use a variety of methods.

[0138] In one possible implementation, the first data collected by the sensor includes: image information of a first user, which includes first lip image information and first body language information. Determining that the first user intends to sing includes: determining that the first user intends to sing when the first user makes a first movement based on the first body language information, or when the first lip shape is identified based on the first lip image information. This ensures more accurate recognition of the user's singing intention.

[0139] Optionally, the first body language information may include, but is not limited to, the first user's head movements, hand gestures, and facial expression changes, etc.

[0140] Optionally, recognizing the first user's first action based on the first body language information may include: recognizing the first user nodding, or recognizing the first user clapping or snapping their fingers.

[0141] Optionally, the first lip shape can be identified based on the first lip image information, which may be to identify that the lip shape of the first user is consistent with the lip shape of the original singer of the song.

[0142] In one possible implementation, method 300 further includes: disabling the singing mode if no second voice information from the first user is detected within a first preset duration, or if the second voice information does not match the text content or music waveform corresponding to the first song within the first preset duration. Thus, after detecting that the user has no intention to sing, the vehicle can control the singing mode to be disabled. This automated management not only makes the in-vehicle entertainment system more intelligent, convenient, and efficient, but also contributes to energy conservation and environmental protection.

[0143] In one embodiment, the first data collected by the sensor includes: third voice information emitted by the first user. Before activating the singing mode, method 300 further includes: controlling the sound-emitting device to play a second song. Then, in step S302, determining that the first user intends to sing based on the first data includes: determining that the first user intends to sing if the text content corresponding to the third voice information matches the text content corresponding to the second song. In this way, the singing intention of the first user can be accurately and efficiently identified, thereby activating the singing function.

[0144] Optionally, matching the text content corresponding to the third voice information with the text content corresponding to the second song includes: the matching degree between the text content corresponding to the third voice information and the text content corresponding to the second song is greater than or equal to a preset threshold.

[0145] In one possible implementation, before determining that the first user intends to sing, method 300 further includes determining that the waveform of the third voice information matches the waveform of the second song. This allows for a more accurate identification of the first user's intention to sing, thereby satisfying the impromptu singing needs of users inside the vehicle.

[0146] Optionally, matching the waveform of the third voice information with the music waveform of the second song includes: the matching degree between the waveform of the third voice information and the music waveform of the second song is greater than or equal to a preset threshold.

[0147] In one embodiment, the first data collected by the sensor includes: image information of a first user, and the image information of the first user includes: second lip image information. Before step S302, method 300 further includes: controlling the sound-generating device to play a second song; then, in step S302, determining that the first user has the intention to sing based on the data includes: determining that the first user has the intention to sing if the second text content identified based on the second lip image information matches the text content corresponding to the second song. In this way, the singing intention of the first user can be accurately and efficiently identified, thereby activating the singing function.

[0148] Optionally, the second text content matches the text content corresponding to the second song, including: the matching degree between the second text content and the text content corresponding to the second song is greater than or equal to a preset threshold.

[0149] In one possible implementation, method 300 further includes: if no fourth voice information emitted by the first user is detected within a second preset time period, or if the fourth voice information is detected within the second preset time period and does not match the text content or music waveform corresponding to the second song, then the singing mode is turned off.

[0150] It should be understood that, in the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0151] Figure 4 is a schematic flowchart of another control method provided in an embodiment of this application. The execution subject of method 400 can be a vehicle. When the execution subject of method 400 is vehicle 100, it can be executed by computing platform 130 in vehicle 100, or by system-on-chip (SoC) in computing platform 130, or by processor in computing platform 130. Method 400 is described below with vehicle as the execution subject. Method 400 can include steps S401 to S402.

[0152] S401, acquire the first data collected by the sensors in the cockpit.

[0153] The aforementioned first data includes: image information of the first user, which includes: first lip image information and first body language information.

[0154] S402, if the first user makes a first action based on the first body language information, or the first lip shape is identified based on the first lip image information, it is determined that the first user has the intention to sing.

[0155] Optionally, the first body language information may include, but is not limited to, the first user's head movements, hand gestures, and facial expression changes, etc.

[0156] Optionally, recognizing the first user's first action based on the first body language information may include: recognizing the first user nodding, or recognizing the first user clapping or snapping their fingers.

[0157] Optionally, the first lip shape can be identified based on the first lip image information, which may be to identify that the lip shape of the first user is consistent with the lip shape of the original singer of the song.

[0158] In one possible implementation, before step S401, the vehicle control sound device plays a second song, and the first data also includes third voice information issued by the first user. Then, before step S402, method 400 further includes: determining that the third voice information matches the text content or music waveform corresponding to the second song.

[0159] In one possible implementation, before step S402, method 400 further includes: determining that the text content corresponding to the first lip image information matches the text content corresponding to the second song.

[0160] S403, activate singing mode.

[0161] In this embodiment, the system can identify the first user's intention to sing based on the first user's body language information or lip image information, and control the singing mode to be turned on. This can quickly meet the impromptu singing needs of users in the car and improve the user's human-computer interaction experience.

[0162] In one possible implementation, before step S403, method 400 further includes: a control prompting device prompting the activation of the singing mode.

[0163] In one possible implementation, in step S403, activating the singing mode includes: activating the singing mode of the first area selected by the first user, or activating the singing mode of all areas in the cabin.

[0164] In one possible implementation, the first data further includes first voice information emitted by the first user. Before step S402, the vehicle determines the first song based on the text content corresponding to the first voice information and the text content corresponding to the first lip image information. Then, after step S403, method 400 further includes controlling the sound-generating device to play a third song. In this way, the user can trigger song playback with simple lip movements and voice, without complex operations or device interactions, ensuring that the user's singing needs are met instantly.

[0165] Optionally, after step S403, method 400 further includes: if no second voice information issued by the first user is detected within a first preset time period, or if the second voice information is detected within the first preset time period and does not match the text content or music waveform corresponding to the first song, then the singing mode is turned off.

[0166] In one possible implementation, before step S401, the vehicle control sound device plays a second song, and the first data also includes a third voice message issued by the first user. Then, after step S403, method 400 further includes: if a fourth voice message issued by the first user is not detected within a second preset time period, or if the fourth voice message is detected within the second preset time period and does not match the text content or music waveform corresponding to the second song, the singing mode is turned off.

[0167] When multiple users are in the cabin, the vehicle can also take into account the singing needs of different users and turn on or off the singing mode in specific areas.

[0168] In one possible implementation, when the vehicle detects a second user in the cabin who requires a do-not-disturb mode, the vehicle can keep the singing mode in the second area of ​​the cabin (where the second user is located) either off or off. In this way, the onboard microphone in the second area will not pick up the second user's voice, and the audio system in the second area will not play the first user's singing.

[0169] For example, the first region is the region corresponding to pitch 1 in Figure 2, and the second region is the region corresponding to pitch 2 in Figure 2.

[0170] Optionally, detecting that the second user is in a state of do-not-disturbance includes detecting that the second user is in a state of rest, watching videos, or playing games.

[0171] In one possible implementation, after the vehicle activates the singing mode in the first area based on the user's selection, and then detects, based on acquired second data, that a third user in the cabin intends to sing, the vehicle can either activate the singing mode for the entire vehicle or activate the singing mode in the third area of ​​the cabin, where the third user is located. This better caters to the singing needs of all users in the vehicle, thus providing a comfortable singing environment for everyone.

[0172] For example, the first region is the region corresponding to pitch 1 in Figure 2, and the third region is the region corresponding to pitch 3 in Figure 2.

[0173] Optionally, if no music is playing in the vehicle, and the vehicle detects that the first user is singing song 1 based on the first data, and the vehicle detects that the third user is singing song 2 based on the second data, the vehicle can control the prompting device to prompt the user (the first user or the third user) to choose to sing song 1 or song 2.

[0174] It should be understood that, in the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0175] Figure 5 is a schematic flowchart of another control method provided in the embodiments of this application. Method 500 may be a detailed description of each step in method 300 or method 400. Method 500 may include steps S501 to S511.

[0176] S501: The system detected that the user has authorized the automatic recommendation function for singing mode.

[0177] For example, a pop-up window can be displayed on the vehicle display screen 202 to prompt the user whether to enable the automatic recommendation function for singing mode. When the user clicks the "Enable" control on the display screen 202, step S502 can be performed; otherwise, method 500 can be terminated.

[0178] For example, users can find the "Settings" function on the in-vehicle display screen 202, and find the corresponding option in the "Settings" function to enable the automatic recommendation function for singing mode.

[0179] S502, a music application (App) is detected playing music.

[0180] S503 initiates the acoustic detection algorithm and continuously detects the fit of the sound waveform.

[0181] S504 determines whether the waveform of the music and the waveform of the user's voice are highly similar.

[0182] Specifically, when the fitting degree between the waveform of the music and the waveform of the user's speech is greater than or equal to the first threshold, it can be considered that the waveform of the music and the waveform of the user's speech are highly fitted, and then step S505 can be performed; otherwise, step S503 can be performed. The aforementioned user speech can be the third speech information in method 300 and method 400, and the music can be the music corresponding to the second song in method 300 and method 400.

[0183] The S505 can capture the text corresponding to the user's voice and obtain the lyrics of the currently playing music from the music app.

[0184] Among them, sound pickup can be understood as the vehicle's audio system capturing and recording the user's voice, and converting the user's voice into corresponding text.

[0185] S506 determines whether the text corresponding to the user's voice matches the lyrics.

[0186] Specifically, when the matching degree between the text corresponding to the user's voice and the lyrics is greater than or equal to the second threshold, it can be considered that the text corresponding to the user's voice matches the content of the lyrics, and then step S507 can be performed; otherwise, it can be considered that the text corresponding to the user's voice does not match the content of the lyrics, and step S504 can be performed.

[0187] For example, if a user's speech is recognized as containing 10 characters and the lyrics contain 8 characters, then when 6 characters in the user's speech match 6 characters in the lyrics, it can be considered that the text content corresponding to the user's speech matches the content of the lyrics, i.e., the second threshold is 6 at this time.

[0188] The S507 prompts the user whether to enable the singing mode for the corresponding vocal range or the full car singing mode.

[0189] For example, the vehicle can display a pop-up window on the in-vehicle display screen 202 to prompt the user to enable the singing mode of the corresponding vocal range, enable the whole vehicle singing mode, or disagree to enable the singing mode; when the user clicks the control on the display screen 202 to enable the singing mode of the corresponding vocal range, enable the whole vehicle singing mode, or disagree to enable the singing mode, step S508 is performed.

[0190] The corresponding pitch range can be understood as at least one pitch range in Figure 2, and the whole car singing mode can be the singing mode of all areas in the cabin in methods 300 and 400.

[0191] S508 determines whether the user agrees to enable the singing mode for the corresponding vocal range or the singing mode for the entire car.

[0192] For example, when the vehicle detects that the user clicks the "Disagree to enable" control, the vehicle may no longer recommend enabling the singing mode during this driving trip. When the vehicle detects that the user clicks the "Enable singing mode for the corresponding vocal range" control, step S509a can be performed. When the vehicle detects that the user clicks the "Enable singing mode for the whole vehicle" control, step S509b can be performed.

[0193] S509a, activate the singing mode for the corresponding vocal range.

[0194] For example, when the vehicle detects that the user clicks the control to enable the singing mode of zone 1, the vehicle can enable the singing mode corresponding to zone 1.

[0195] S509b, turn on the car's singing mode.

[0196] For example, when the vehicle detects that the user clicks to start the singing mode for the whole vehicle, the vehicle can start the singing mode in zone 1 to zone 4.

[0197] S510a, detects that the voice and music waveforms in the corresponding audio range do not match, or detects that no user voice was recorded within a preset duration.

[0198] For example, when the vehicle activates the singing mode in zone 1, it detects that the waveform of the user's voice from zone 1 and the waveform of the music have a fitting degree less than a third threshold, or that no user's voice from zone 1 is recorded within 5 minutes.

[0199] S510b detects that the voice and music waveforms do not match in all audio regions, or detects that no user voice was recorded within a preset duration.

[0200] For example, when the vehicle turns on the full-vehicle singing mode, if the waveform of the user's voice and the waveform of the music are less than a third threshold, or if no user's voice is recorded in the vehicle within 5 minutes.

[0201] S511, a pop-up window reminds the user to turn off the singing mode.

[0202] For example, the vehicle can display a pop-up window on the in-vehicle display screen 202 to prompt the user to turn off the singing mode. When the vehicle detects that the user clicks the "Turn off singing mode" control on the display screen 202, the vehicle can turn off the singing mode; otherwise, the vehicle can keep the singing mode on.

[0203] It should be noted that in method 500, the interaction between the vehicle and the user can be achieved through a human-computer interaction assistant (e.g., a voice assistant).

[0204] In this embodiment, when music is played in the car, the user's intention to sing can be determined through acoustic detection algorithms and the matching degree between the music lyrics and the text corresponding to the user's voice. After determining that the user wants to sing, the user can be recommended to turn on the singing mode. In this way, when users in the car want to sing along or duet while the music is playing, they do not need to manually turn on the singing mode, which can quickly meet the impromptu karaoke needs of users in the car and improve the user's human-computer interaction experience.

[0205] Figure 6 is a schematic flowchart of another control method provided in the embodiments of this application. Method 600 may be a detailed description of each step in method 300 or method 400. Method 600 may include steps S601 to S611.

[0206] S601, detected that the user has authorized the automatic recommendation function for singing mode.

[0207] For example, a pop-up window can be displayed on the vehicle display screen 202 to prompt the user whether to enable the automatic recommendation function for singing mode. When the user clicks the "Enable" control on the display screen 202, step S602 is performed; otherwise, method 600 can be terminated.

[0208] For example, users can find the "Settings" function on the in-vehicle display screen 202, and find the corresponding option in the "Settings" function to enable the automatic recommendation function for singing mode.

[0209] S602, a music app is detected playing music.

[0210] S603, activates the acoustic detection algorithm and continuously detects the fit of the sound waveform.

[0211] S604 determines whether the waveform of the music and the waveform of the user's voice are highly similar.

[0212] Specifically, when the fitting degree between the waveform of the music and the waveform of the user's speech is greater than or equal to the first threshold, it can be considered that the waveform of the music and the waveform of the user's speech are highly fitted, and then step S605 can be performed; otherwise, step S603 can be performed. The aforementioned user speech can be the third speech information in method 300 and method 400, and the music can be the music corresponding to the second song in method 300 and method 400.

[0213] S605 initiates visual algorithms to detect the user's lip movements to infer text, picks up the user's voice to obtain the text corresponding to the voice, and retrieves the lyrics of the currently playing music from the music app.

[0214] Optionally, the in-vehicle camera 205 can be activated to acquire the user's image information, and the user's image information can be input into a visual detection algorithm to infer lyrics based on the user's lip movements. Further, optionally, to improve the accuracy of the visual algorithm's output, facial or lip image information (which could be the second lip image information in method 300) can be extracted from the user's image information and input into the visual detection algorithm. Alternatively, the in-vehicle camera 205 can be used to directly acquire the user's facial or lip image information, and then the user's facial and lip information can be input into the visual detection algorithm.

[0215] S606, a multimodal sensor for determining whether a user is singing.

[0216] Specifically, if the matching degree between the text corresponding to the user's voice and the lyrics is greater than or equal to the second threshold, and the matching degree between the text and the lyrics detected by the visual algorithm is greater than or equal to the fourth threshold, it can be determined that the user is singing, and then step S607 is performed; otherwise, step S604 can be performed.

[0217] Alternatively, user voice and lip-reading image information can be directly input into a large model to determine whether the user is singing.

[0218] The S607 prompts the user whether to enable the singing mode for the corresponding vocal range or the full car singing mode.

[0219] For example, the vehicle can display a pop-up window on the in-vehicle display screen 202 to prompt the user to enable the singing mode of the corresponding vocal range, enable the whole vehicle singing mode, or disagree to enable the singing mode; when the user clicks the control on the display screen 202 to enable the singing mode of the corresponding vocal range, enable the whole vehicle singing mode, or disagree to enable the singing mode, step S608 is performed.

[0220] The corresponding pitch range can be understood as at least one pitch range in Figure 2, and the whole car singing mode can be the singing mode of all areas in the cabin in methods 300 and 400.

[0221] S608 determines whether the user agrees to enable the singing mode for the corresponding vocal range or the singing mode for the entire car.

[0222] For example, when the vehicle detects that the user clicks the "Disagree to enable" control, the vehicle may no longer recommend enabling the singing mode during this driving trip. When the vehicle detects that the user clicks the "Enable singing mode for the corresponding vocal range" control, step S609a can be performed. When the vehicle detects that the user clicks the "Enable singing mode for the whole vehicle" control, step S609b can be performed.

[0223] S609a, activate the singing mode for the corresponding vocal range.

[0224] For example, when the vehicle detects that the user clicks the control to enable the singing mode of zone 1, the vehicle can enable the singing mode corresponding to zone 1.

[0225] S609b, activate the car's singing mode.

[0226] For example, when the vehicle detects that the user clicks to start the singing mode for the whole vehicle, the vehicle can start the singing mode in zone 1 to zone 4.

[0227] S610a, detects that the voice and music waveforms in the corresponding audio range do not match, or detects that no user voice was recorded within a preset duration.

[0228] For example, when the vehicle turns on the singing mode of zone 1, it detects that the waveform of the user's voice from zone 1 and the waveform of the music have a fitting degree of less than a third threshold, or that no user's voice from zone 1 is recorded within 6 minutes.

[0229] S610b detects that the voice and music waveforms do not match in all audio regions, or detects that no user voice was recorded within a preset duration.

[0230] For example, when the vehicle turns on the full-vehicle singing mode, if the waveform of the user's voice and the waveform of the music are less than a third threshold, or if no user's voice is recorded in the vehicle within 6 minutes.

[0231] S611, a pop-up window reminds the user to turn off the singing mode.

[0232] For example, the vehicle can display a pop-up window on the in-vehicle display screen 202 to prompt the user to turn off the singing mode. When the vehicle detects that the user clicks the "Turn off singing mode" control on the display screen 202, the vehicle can turn off the singing mode; otherwise, the vehicle can keep the singing mode on.

[0233] It should be noted that in method 600, the interaction between the vehicle and the user can be achieved through a human-computer interaction assistant (e.g., a voice assistant).

[0234] In this embodiment, when music is played in the car, the user's intention to sing can be determined through acoustic detection algorithms, visual algorithms, and the matching degree between the music lyrics and the text corresponding to the user's voice. After determining that the user wants to sing, the user can be recommended to turn on the singing mode. In this way, when users in the car want to sing along or duet while the music is playing, they do not need to manually turn on the singing mode, which can quickly meet the impromptu karaoke needs of users in the car and improve the user's human-computer interaction experience.

[0235] Figure 7 is a schematic flowchart of another control method provided in the embodiments of this application. Method 700 may be a detailed description of each step in method 300 or method 400. Method 700 may include steps S701 to S711 and steps S721 to S726.

[0236] S701 detected that the user has authorized the automatic recommendation function for singing mode.

[0237] For example, a pop-up window can be displayed on the vehicle display screen 202 to prompt the user whether to enable the automatic recommendation function for singing mode. When the user clicks the "Enable" control on the display screen 202, step S702 is performed; otherwise, method 700 can be terminated.

[0238] For example, users can find the "Settings" function on the in-vehicle display screen 202, and find the corresponding option in the "Settings" function to enable the automatic recommendation function for singing mode.

[0239] Optionally, step S701 further includes: detecting that the user has authorized the activation of the music recommendation function when singing a song.

[0240] S702 determines whether a music app is playing music.

[0241] For example, if it is detected that a music app is playing music, step S703 can be performed; otherwise, step S721 can be performed.

[0242] S703 initiates the acoustic detection algorithm and continuously detects the fit of the sound waveform.

[0243] S704 determines whether the waveform of the music and the waveform of the user's voice are highly similar.

[0244] Specifically, when the fitting degree between the waveform of the music and the waveform of the user's speech is greater than or equal to a first threshold, it can be considered that the waveform of the music and the waveform of the user's speech are highly fitted, and step S705 can be performed; otherwise, step S703 can be performed. The aforementioned user speech can be the first speech information in methods 300 and 400, and the music can be the music corresponding to the first song in methods 300 and 400.

[0245] The S705 can capture the text corresponding to the user's voice and obtain the lyrics of the currently playing music from the music app.

[0246] Optionally, step S705 further includes: activating a visual algorithm to detect the user's lip movements and infer text.

[0247] Optionally, the in-vehicle camera 205 can be activated to acquire the user's image information, and the user's image information can be input into a visual detection algorithm to infer lyrics based on the user's lip movements and facial muscle changes. Further, optionally, to improve the accuracy of the visual algorithm's output, facial image information or lip image information (which can correspond to the first lip image information in method 300) can be extracted from the user's image information and input into the visual detection algorithm. Alternatively, the user's facial information or lip information can be directly acquired through the in-vehicle camera 205, and then the user's facial and lip information can be input into the visual detection algorithm.

[0248] S706, a multimodal sensor for determining whether a user is singing.

[0249] Specifically, if the matching degree between the text corresponding to the user's voice and the lyrics is greater than or equal to the second threshold, and / or the matching degree between the text and the lyrics detected by the visual algorithm is greater than or equal to the third threshold, it can be determined that the user is singing, then proceed to step S707; otherwise, proceed to step S704.

[0250] Alternatively, user voice and lip-reading image information can be directly input into a large model to determine whether the user is singing.

[0251] The S707 prompts the user whether to enable the singing mode for the corresponding vocal range or the full car singing mode.

[0252] For example, the vehicle can display a pop-up window on the in-vehicle display screen 202 to prompt the user to enable the singing mode of the corresponding vocal range, enable the whole vehicle singing mode, or disagree to enable the singing mode; when the user clicks the control on the display screen 202 to enable the singing mode of the corresponding vocal range, enable the whole vehicle singing mode, or disagree to enable the singing mode, step S708 is performed.

[0253] S708 determines whether the user agrees to enable the singing mode for the corresponding vocal range or the singing mode for the entire car.

[0254] For example, when the vehicle detects that the user clicks the "Disagree to enable" control, the vehicle may no longer recommend enabling the singing mode during this driving trip. When the vehicle detects that the user clicks the "Enable singing mode for the corresponding vocal range" control, step S709a can be performed. When the vehicle detects that the user clicks the "Enable singing mode for the whole vehicle" control, step S709b can be performed.

[0255] S709a, activate the singing mode for the corresponding vocal range and play music.

[0256] For example, when the vehicle detects that the user clicks the control to enable the singing mode of zone 1, the vehicle can enable the singing mode corresponding to zone 1.

[0257] S709b, activate the car's karaoke mode and play music.

[0258] For example, when the vehicle detects that the user clicks to start the singing mode for the whole vehicle, the vehicle can start the singing mode in zone 1 to zone 4.

[0259] S710a detects that the voice and music waveforms in the corresponding audio range do not match, or detects that no user voice was recorded within a preset duration.

[0260] For example, when the vehicle activates the singing mode in zone 1, it detects that the waveform of the user's voice from zone 1 and the waveform of the music have a fitting degree less than a third threshold, or that no user's voice from zone 1 is recorded within 5 minutes.

[0261] S710b detects that the voice and music waveforms do not match in all audio regions, or detects that no user voice was recorded within a preset duration.

[0262] For example, when the vehicle turns on the full-vehicle singing mode, if the waveform of the user's voice and the waveform of the music are less than a third threshold, or if no user's voice is recorded in the vehicle within 5 minutes.

[0263] S711, a pop-up window reminds the user to turn off the singing mode.

[0264] For example, the vehicle can display a pop-up window on the in-vehicle display screen 202 to prompt the user to turn off the singing mode. When the vehicle detects that the user clicks the "Turn off singing mode" control on the display screen 202, the vehicle can turn off the singing mode; otherwise, the vehicle can keep the singing mode on.

[0265] S721, initiate a visual algorithm to detect the user's body language information and / or lip image information.

[0266] Optionally, the in-vehicle camera 205 can be activated to obtain the user's image information, and then the user's body language information and / or lip image information can be obtained from the user's image information.

[0267] S722 determines whether the user is singing.

[0268] For example, if the user makes a first action based on body language information or a first lip shape is identified based on lip image information, it can be determined that the user is singing. In this case, step S723 can be performed; otherwise, step S721 can be performed.

[0269] Optionally, body language information may include, but is not limited to, the user's head movements, hand gestures, and facial expression changes.

[0270] Optionally, recognizing a user's first action based on body language information may include recognizing a user nodding, or recognizing a user clapping or snapping their fingers.

[0271] Optionally, the first lip shape can be identified based on the lip image information, which may be to identify that the user's lip shape matches the lip shape of the original singer of the song.

[0272] S723 initiates voice pickup to obtain the text corresponding to the user's voice, and uses a visual algorithm to infer the text content of the user's voice.

[0273] S724, multimodal fusion detection of lyrics, obtains the name of the song the user is singing.

[0274] Specifically, lyrics can be detected by combining the text corresponding to the user's voice obtained through sound pickup and the inference of the text content of the user's voice using visual algorithms, thereby obtaining the name of the song the user is singing.

[0275] Alternatively, user voice and lip-reading image information can be directly input into a large model to determine whether the user is singing.

[0276] S725 determines whether the corresponding song name has been found.

[0277] Specifically, if the lyrics obtained from multimodal fusion detection lead to a corresponding song name, then step S726 can be performed; otherwise, step S721 can be performed.

[0278] The S726 displays a pop-up window reminding the user whether to play the corresponding song and whether to enable the corresponding vocal range or the car's karaoke mode.

[0279] It should be noted that in method 700, the interaction between the vehicle and the user can be achieved through a human-computer interaction assistant (e.g., a voice assistant).

[0280] In this embodiment, when music is playing in the car, the user's intention to sing can be determined by the degree of matching between the lyrics and the text corresponding to the user's voice. Once it is determined that the user wants to sing, the system can recommend that the user activate the singing mode. When no music is playing in the car, the system can search for the song the user is currently singing using acoustic and visual algorithms, and recommend playing the corresponding music and activating the singing mode. This approach quickly satisfies the impromptu karaoke needs of users in the car and improves the user's human-computer interaction experience.

[0281] Figure 8 is a schematic flowchart of another control method provided in the embodiments of this application. Method 800 may be a detailed description of each step in method 300 or method 400. Method 800 may include steps S801 to S811 and steps S821 to S826.

[0282] S801: The system detected that the user has authorized the automatic recommendation function for singing mode.

[0283] For example, a pop-up window can be displayed on the vehicle display screen 202 to prompt the user whether to enable the automatic recommendation function for singing mode. When the user clicks the "Enable" control on the display screen 202, step S802 is performed; otherwise, method 800 can be terminated.

[0284] For example, users can find the "Settings" function on the in-vehicle display screen 202, and find the corresponding option in the "Settings" function to enable the automatic recommendation function for singing mode.

[0285] Optionally, step S801 further includes: detecting that the user has authorized the activation of the music recommendation function when singing a song.

[0286] S802 determines whether a music app is playing music.

[0287] For example, if music is detected to be playing in a music app, step S803 can be performed; otherwise, step S821 can be performed.

[0288] S803 initiates the acoustic detection algorithm and continuously detects the fit of the sound waveform.

[0289] S804 determines whether the waveform of the music and the waveform of the user's voice are highly similar.

[0290] Specifically, when the fitting degree between the waveform of the music and the waveform of the user's speech is greater than or equal to a first threshold, it can be considered that the waveform of the music and the waveform of the user's speech are highly fitted, and step S805 can be performed; otherwise, step S803 can be performed. The aforementioned user speech can be the first speech information in methods 300 and 400, and the music can be the music corresponding to the first song in methods 300 and 400.

[0291] The S805 can capture the text corresponding to the user's voice and obtain the lyrics of the currently playing music from the music app.

[0292] Optionally, step S805 further includes: activating a visual algorithm to detect the user's lip movements and infer text.

[0293] Optionally, the in-vehicle camera 205 can be activated to acquire the user's image information, and this image information can be input into a visual detection algorithm to infer lyrics based on the user's lip movements and facial muscle changes. Further, optionally, to improve the accuracy of the visual algorithm's output, facial or lip image information (which can correspond to the first lip image information in method 300 or the third lip image information in method 400) can be extracted from the user's image information and input into the visual detection algorithm. Alternatively, the user's facial or lip information can be directly acquired through the in-vehicle camera 205, and then input into the visual detection algorithm.

[0294] S806, a multimodal sensor for determining whether a user is singing.

[0295] Specifically, if the matching degree between the text corresponding to the user's voice and the lyrics is greater than or equal to the second threshold, and / or the matching degree between the text and the lyrics detected by the visual algorithm is greater than or equal to the third threshold, it can be determined that the user is singing, then proceed to step S807; otherwise, proceed to step S804.

[0296] Alternatively, user voice and lip-reading image information can be directly input into a large model to determine whether the user is singing.

[0297] The S807 prompts the user whether to enable the singing mode for the corresponding vocal range or the full car singing mode.

[0298] For example, the vehicle can display a pop-up window on the in-vehicle display screen 202 to prompt the user to enable the singing mode of the corresponding vocal range, enable the whole vehicle singing mode, or disagree to enable the singing mode; when the user clicks the control on the display screen 202 to enable the singing mode of the corresponding vocal range, enable the whole vehicle singing mode, or disagree to enable the singing mode, step S808 is performed.

[0299] S808 determines whether the user agrees to enable the singing mode for the corresponding vocal range or the singing mode for the entire car.

[0300] For example, when the vehicle detects that the user clicks the "Disagree to enable" control, the vehicle may no longer recommend enabling the singing mode during this driving trip. When the vehicle detects that the user clicks the "Enable singing mode for the corresponding vocal range" control, step S809a can be performed. When the vehicle detects that the user clicks the "Enable singing mode for the whole vehicle" control, step S809b can be performed.

[0301] S809a, activate the singing mode for the corresponding vocal range and play music.

[0302] For example, when the vehicle detects that the user clicks the control to enable the singing mode of zone 1, the vehicle can enable the singing mode corresponding to zone 1.

[0303] S809b, activate the car's karaoke mode and play music.

[0304] For example, when the vehicle detects that the user clicks to start the singing mode for the whole vehicle, the vehicle can start the singing mode in zone 1 to zone 4.

[0305] S810a detects that the voice and music waveforms in the corresponding audio range do not match, or detects that no user voice was recorded within a preset duration.

[0306] For example, when the vehicle activates the singing mode in zone 1, it detects that the waveform of the user's voice from zone 1 and the waveform of the music have a fitting degree less than a third threshold, or that no user's voice from zone 1 is recorded within 5 minutes.

[0307] S810b detects that the voice and music waveforms do not match in all audio regions, or detects that no user voice was recorded within a preset duration.

[0308] For example, when the vehicle turns on the full-vehicle singing mode, if the waveform of the user's voice and the waveform of the music are less than a third threshold, or if no user's voice is recorded in the vehicle within 5 minutes.

[0309] S811, a pop-up window reminds the user to turn off the singing mode.

[0310] For example, the vehicle can display a pop-up window on the in-vehicle display screen 202 to prompt the user to turn off the singing mode. When the vehicle detects that the user clicks the "Turn off singing mode" control on the display screen 202, the vehicle can turn off the singing mode; otherwise, the vehicle can keep the singing mode on.

[0311] S821, activates voice pickup to acquire voice information sent by the user.

[0312] S822 determines whether the user is singing.

[0313] For example, when the waveform of the user's spoken language matches a preset sound waveform, it can be determined that the user is singing.

[0314] Optionally, the preset sound can be music or the human voice corresponding to the trigger keyword.

[0315] S823 initiates voice pickup to obtain the text corresponding to the user's voice, and uses a visual algorithm to infer the text content of the user's voice.

[0316] S824, multimodal fusion detection of lyrics, obtains the name of the song the user is singing.

[0317] Specifically, lyrics can be detected by combining the text corresponding to the user's voice obtained through sound pickup and the inference of the text content of the user's voice using visual algorithms, thereby obtaining the name of the song the user is singing.

[0318] S825, determine whether the corresponding song name has been found.

[0319] Specifically, if the lyrics obtained from multimodal fusion detection lead to a corresponding song name, then step S826 can be performed; otherwise, step S821 can be performed.

[0320] The S826 pops up a reminder to the user whether to play the corresponding song and whether to enable the corresponding vocal range or the car's karaoke mode.

[0321] It should be noted that in method 800, the interaction between the vehicle and the user can be achieved through a human-computer interaction assistant (e.g., a voice assistant).

[0322] In this embodiment, when music is playing in the car, the user's intention to sing can be determined by the degree of matching between the lyrics and the text corresponding to the user's voice. Once it is determined that the user wants to sing, the user can be recommended to start the singing mode. When no music is playing in the car, acoustic detection algorithms and visual algorithms can be used to search for the song the user is currently singing, and the corresponding music and singing mode can be recommended to the user. This approach can quickly meet the impromptu karaoke needs of users in the car and improve the user's human-computer interaction experience.

[0323] Figures 9 to 15 are schematic diagrams of an application scenario to which the control method provided in the embodiments of this application is applicable. Methods 300 to 600 can be applied to this application scenario.

[0324] As shown in Figure 9(a), the vehicle's central control screen displays interface 900 and function bar 910. Interface 900 includes user account login information 901 (the vehicle is not currently logged into), a Bluetooth icon 902, a Wi-Fi icon 903, a cellular network signal icon 904, a vehicle map application search box 905, a card to switch to displaying all installed applications in the vehicle 906, a card to switch to displaying the vehicle music application 907, a display card for the vehicle's remaining battery power and remaining driving range 908, and a display card for the vehicle's 360-degree (°) surround view function 909. The vehicle map application search box 905 may include user-defined controls for going home 9051 and going to work 9052. The function bar 910 includes an icon 911 for switching to the central control screen desktop, an icon 912 for vehicle recirculation, an icon 913 for driver's seat heating function, an icon 914 for driver's area air conditioning temperature display, an icon 915 for passenger's area air conditioning temperature display, an icon 916 for passenger's seat heating function, and a volume setting icon 917. When the vehicle detects that the user clicks on the card 907 of the in-vehicle music application, the graphical user interface (GUI) shown in Figure 9(b) can be displayed on the vehicle's central control screen.

[0325] As shown in Figure 9(b), the GUI includes a prompt box 918 to prompt the user whether to enable the automatic recommendation function for singing mode. When the vehicle detects that the user clicks the "agree" control in the prompt box 918, the vehicle can enable the automatic recommendation function for singing mode.

[0326] As shown in Figure 10, when a user utters the voice message "C xxxxxx D" while playing music, the vehicle can pick up the corresponding text information through sound pickup. Simultaneously, the vehicle can activate its in-car camera to obtain the user's image information as shown in Figure 11. After processing this image information, the user's lip image information can be obtained. As shown in Figure 12, the user's lip image information can be input into a visual algorithm to infer whether the user is singing and the specific content of the user's voice message. On the other hand, as shown in Figure 13(a), the GUI interface 919 displays the song "Song B" that the user is currently listening to. The vehicle can obtain the lyrics "C xxxxxx D" of the currently playing song "Song B" through a music app. Based on the above detection results, the GUI shown in Figure 13(b) can be displayed on the vehicle's central control screen.

[0327] As shown in Figure 13(b), the GUI includes a prompt box 920 that prompts the user, "We have detected that you are singing. Do you want to enable the car-wide singing mode?" When the vehicle detects that the user clicks the "Enable" control in the prompt box 920, as shown in Figure 14, the car-wide singing mode is activated. All users in the car can use the in-car microphone instead of an external microphone and enjoy karaoke anytime, anywhere through the surround sound of the car's audio system. When the vehicle detects that no user has made a sound within 5 minutes, the GUI shown in Figure 15 can be displayed on the vehicle's central control screen.

[0328] As shown in Figure 15, the GUI may include a prompt box 921 to prompt the user in the vehicle whether to turn off the car-sharing mode if the user has not made any sound within 5 minutes. When the vehicle detects that the user clicks the "off" control in the prompt box 921, the vehicle can turn off the car-sharing mode.

[0329] Figures 16 to 19 are schematic diagrams of an application scenario to which the control method provided in the embodiments of this application is applicable. Methods 300 to 600 can be applied to this application scenario.

[0330] The vehicle's central control screen displays the GUI shown in Figure 16(a), which shows the song "Song D" currently playing in the vehicle, with lyrics "E xxxxxxx F". When the vehicle detects that a user inside the vehicle is singing the song with lyrics "E xxxxxxx F" using sound pickup and / or visual algorithms, the central control screen can display the GUI shown in Figure 16(b).

[0331] As shown in Figure 16(b), the GUI includes a prompt box 1002 to prompt the user "It is detected that you are singing. Do you want to enable the singing mode for a partial range?" When the vehicle detects that the user has clicked the "Enable" control in the prompt box 1002, the GUI shown in Figure 17 can be displayed on the vehicle's central control screen.

[0332] As shown in Figure 17, the GUI may include a prompt box 1003 to prompt the user to activate the corresponding vocal zone for the singing mode. The prompt box 1003 includes controls for vocal zones 1 through 4. When the vehicle detects that the user clicks the "Vocal Zone 2" control in the prompt box 1003, as shown in Figure 18, the vehicle can activate the singing mode for vocal zone 2. The user in vocal zone 2 can use the in-vehicle microphone instead of an external microphone and enjoy karaoke anytime, anywhere in the car through the surround sound of the in-vehicle audio system. When the vehicle detects that the user in vocal zone 2 has not made a sound for 5 minutes, the GUI shown in Figure 19 can be displayed on the vehicle's central control screen.

[0333] As shown in Figure 19, the GUI includes a dialog box 1004 and a voice assistant 1005. The voice assistant 1005 can broadcast the contents of the dialog box 1004 to the user via voice, that is, prompt the user via voice, "The vehicle has detected that the user in zone 2 has not made a sound within 5 minutes. Do you want to turn off the singing mode corresponding to zone 2?" When the vehicle detects the driver's voice message "Okay, please turn it off", the vehicle can turn off the singing mode corresponding to zone 2.

[0334] Figures 20 and 21 are schematic diagrams of an application scenario in which the control method provided in the embodiments of this application is applicable. Methods 300, 400, 700 and 800 can be applied to this application scenario.

[0335] As shown in Figure 20, the user, while driving, did not play music and uttered the voice message "G xxxxxx H". At this time, the vehicle can use its voice pickup function to identify the text content corresponding to this voice message. Simultaneously, the vehicle can activate the in-vehicle camera 205 to acquire the user's image information. After processing this image information, the user's lip image information can be obtained. This lip image information is then input into the visual algorithm shown in Figure 21, which can identify the text content of the voice message. Combining the voice pickup function and the text content identified by the visual algorithm, the song title (i.e., song E) corresponding to the text content can be inferred. After determining the song title, the GUI shown in Figure 22(a) can be displayed on the vehicle's central control screen.

[0336] As shown in Figure 22(a), the GUI includes a prompt box 1101 to prompt the user, "It has been detected that you are singing song E. Do you want to play this song and start the singing mode?" After the vehicle detects that the user clicks the "OK" control in the prompt box 1101, the GUI shown in Figure 22(b) can be displayed on the vehicle's central control screen, that is, the playback interface of song E is displayed, and song E is played. At the same time, the vehicle can start the full car singing mode, or, according to the user's selection, start the singing mode of a certain range of notes.

[0337] It should be noted that the application scenarios shown in Figures 9 to 22 are merely illustrative examples and should not be construed as limiting this application. Those skilled in the art can modify and design the human-machine interface between the vehicle and the user according to actual needs.

[0338] It should be understood that, in the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0339] Figure 23 is a schematic diagram of a control device provided in an embodiment of this application. The device 1200 may include an acquisition unit 1210, a storage unit 1220, and a processing unit 1230. The acquisition unit 1210 is used to acquire instructions and / or data; the storage unit 1220 is used to implement the corresponding storage function and store the corresponding instructions and / or data; the processing unit 1230 is used to perform data processing so that the device 1200 implements the aforementioned control method.

[0340] In one embodiment, the device 1200 includes: an acquisition unit 1210 and a processing unit 1230; the acquisition unit 1210 is used to acquire first data collected by sensors in the cockpit; the processing unit 1230 is used to activate a singing mode when it is determined from the first data that the first user intends to sing; the processing unit 1230 is specifically used to activate a singing mode for a first area or a singing mode for the entire cockpit area based on a first area selected by the first user.

[0341] In one possible implementation, the processing unit 1230 is also used to control the prompting device to prompt the activation of the singing mode.

[0342] In one possible implementation, the processing unit 1230 is further configured to keep the singing mode in the second area of ​​the cabin off or off when the second user in the cabin has a do-not-disturb requirement, and the second user is located in the second area.

[0343] In one possible implementation, the processing unit 1230 is specifically used to enable the singing mode in the first area; the processing unit 1230 is also used to enable the singing mode in all areas of the cabin or enable the singing mode in the third area of ​​the cabin when it is determined from the second data that the third user in the cabin intends to sing, and the third user is located in the third area.

[0344] In one possible implementation, the first data includes: image information of the first user and first voice information emitted by the first user. The image information of the first user includes first lip image information. The processing unit 1230 is specifically used to: determine the first song based on the first lip image information and the first voice information; and control the sound-emitting device to play the first song.

[0345] In one possible implementation, the image information of the first user further includes: first body language information. The processing unit 1230 is specifically used to determine that the first user has the intention to sing when the first user makes a first action based on the first body language information, or when the first lip shape is identified based on the first lip image information.

[0346] In one possible implementation, the processing unit 1230 is further configured to close the singing mode if no second voice information issued by the first user is detected within a first preset time period, or if the second voice information is detected within the first preset time period and does not match the text content or music waveform corresponding to the first song.

[0347] In one possible implementation, the first data includes: third voice information emitted by the first user; the processing unit 1230 is also used to control the sound-emitting device to play a second song; the processing unit 1230 is specifically used to determine that the first user has the intention to sing when the text content corresponding to the third voice information matches the text content corresponding to the second song.

[0348] In one possible implementation, the processing unit 1230 is further configured to determine whether the waveform of the third speech information matches the music waveform of the second song.

[0349] In one possible implementation, the first data includes: image information of the first user, the image information of the first user includes: second lip image information, the processing unit 1230 is further used to control the sound-generating device to play the second song; the processing unit 1230 is specifically used to determine that the first user has the intention to sing when the second text content identified based on the second lip image information matches the text content corresponding to the second song.

[0350] In one possible implementation, the processing unit 1230 is further configured to close the singing mode if the fourth voice information emitted by the first user is not detected within a second preset time period, or if the fourth voice information is detected within a second preset time period and does not match the text content or music waveform corresponding to the second song.

[0351] In one embodiment, the device 1200 includes an acquisition unit 1210 and a processing unit 1230; the acquisition unit 1210 is used to acquire first data collected by sensors in the cockpit, the first data including: image information of a first user, the image information of the first user including: first lip image information and first body language information; the processing unit 1230 is used to: determine that the first user has the intention to sing when the first user makes a first action based on the first body language information, or when the first lip shape is identified based on the third lip image information; and activate the singing mode.

[0352] In one possible implementation, the first data further includes first voice information emitted by the first user, and the processing unit 1230 is further configured to: determine a first song based on the first lip image information and the first voice information; and control the sound-emitting device to play the first song.

[0353] In one possible implementation, the processing unit 1230 is further configured to close the singing mode if no second voice information issued by the first user is detected within a first preset time period, or if the second voice information is detected within the first preset time period and does not match the text content or music waveform corresponding to the first song.

[0354] In one possible implementation, the first data also includes third voice information emitted by the first user, and the processing unit 1230 is further configured to: control the sound-emitting device to play the second song; and determine whether the third voice information matches the text content or music waveform corresponding to the second song.

[0355] In one possible implementation, the processing unit 1230 is further configured to determine whether the text content corresponding to the first lip image information matches the text content corresponding to the second song.

[0356] In one possible implementation, the processing unit 1230 is further configured to close the singing mode if the fourth voice information emitted by the first user is not detected within a second preset time period, or if the fourth voice information is detected within a second preset time period and does not match the text content or music waveform corresponding to the second song.

[0357] In one possible implementation, the processing unit 1230 is also used to control the prompting device to prompt the activation of the singing mode.

[0358] In one possible implementation, the processing unit 1230 is specifically used to enable the singing mode of the first area selected by the first user, or to enable the singing mode of all areas of the cabin.

[0359] In one possible implementation, the processing unit 1230 is specifically used to keep the singing mode in the second area of ​​the cabin off or turn it off when the second user in the cabin has a do-not-disturb requirement, and the second user is located in the second area.

[0360] In one possible implementation, the processing unit 1230 is specifically used to enable the singing mode in the first area. The processing unit 1230 is also used to enable the singing mode in all areas of the cabin or enable the singing mode in the third area of ​​the cabin when it is determined from the second data that the third user in the cabin intends to sing. The third user is located in the third area.

[0361] Figure 24 is a schematic diagram of another control device provided in an embodiment of this application.

[0362] The device 1300 includes a memory 1310, a processor 1320, and a communication interface 1330. The memory 1310, processor 1320, and communication interface 1330 are connected via an internal connection path. The memory 1310 stores instructions, and the processor 1320 executes the instructions stored in the memory 1310 to control the communication interface 1330 to acquire information, thereby enabling the device 1300 to implement the aforementioned control method. Optionally, the memory 1310 can be coupled to the processor 1320 via an interface, or it can be integrated with the processor 1320.

[0363] It should be noted that the communication interface 1330 described above uses a transceiver device, such as, but not limited to, a transceiver. The communication interface 1330 may also include an input / output interface.

[0364] The processor 1320 stores one or more computer programs, which include instructions. When the instructions are executed by the processor 1320, the control device 1300 performs the control methods described in the above embodiments.

[0365] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 1320 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1310, and the processor 1320 reads the information in memory 1310 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0366] Optionally, the communication interface 1330 in FIG24 can implement the acquisition unit 1210 in FIG23, the memory 1310 in FIG24 can implement the storage unit 1220 in FIG23, and the processor 1320 in FIG24 can implement the processing unit 1230 in FIG23.

[0367] This application also provides a computer-readable storage medium storing program code that, when run on a computer, causes the computer to execute any of the methods shown in Figures 4 to 8.

[0368] This application also provides a computer program product, which includes a computer program that, when run, causes a computer to perform any of the methods shown in Figures 4 to 8.

[0369] This application also provides a chip, including: a circuit for performing any of the methods in Figures 4 to 8 above.

[0370] This application also provides a vehicle, which includes a control device as shown in FIG23 or FIG24.

[0371] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0372] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0373] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0374] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0375] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0376] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0377] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A control method, characterized in that, The method includes: Acquire the first data collected by sensors inside the cockpit; When it is determined from the first data that the first user intends to sing, the singing mode is activated. The activation of the singing mode includes: activating the singing mode of the first area selected by the first user, or activating the singing mode of all areas of the cabin.

2. The method as described in claim 1, characterized in that, The method further includes: The control prompt device indicates that the singing mode should be turned on.

3. The method as described in claim 1 or 2, characterized in that, The method further includes: When a second user in the cabin has a do-not-disturb requirement, the singing mode in the second area of ​​the cabin will remain either off or off. The second user is located in the second area.

4. The method as described in claim 1 or 2, characterized in that, The method of activating the singing mode includes: activating the singing mode in the first area; the method further includes: When it is determined from the second data that a third user in the cabin intends to sing, the singing mode is activated for all areas of the cabin or for the third area within the cabin, where the third user is located.

5. The method according to any one of claims 1 to 4, characterized in that, The first data includes: image information of the first user and first voice information emitted by the first user, wherein the image information of the first user includes first lip image information, and the method further includes: The first song is determined based on the first lip image information and the first voice information; Control the sound-producing device to play the first song.

6. The method as described in claim 5, characterized in that, The image information of the first user also includes: first body language information, and the step of determining that the first user intends to sing based on the first data includes: If the first user makes a first action based on the first body language information, or if the first lip shape is identified based on the first lip image information, it is determined that the first user has the intention to sing.

7. The method as described in claim 5 or 6, characterized in that, The method further includes: If no second voice message from the first user is detected within the first preset time period, or if the second voice message is detected within the first preset time period and does not match the text content or music waveform corresponding to the first song, the singing mode is turned off.

8. The method according to any one of claims 1 to 4, characterized in that, The first data includes: third voice information emitted by the first user. Before activating the singing mode, the method further includes: Control the sound-producing device to play the second song; The step of determining that the first user intends to sing based on the first data includes: If the text content corresponding to the third voice information matches the text content corresponding to the second song, it is determined that the first user has the intention to sing.

9. The method as described in claim 8, characterized in that, Before determining that the first user has the intention to sing, the method further includes: The waveform of the third voice information is determined to match the music waveform of the second song.

10. The method according to any one of claims 1 to 4, 8 and 9, characterized in that, The first data includes: image information of the first user, the image information of the first user includes: second lip image information, and before activating the singing mode, the method further includes: Control the sound-producing device to play the second song; The step of determining that the first user intends to sing based on the first data includes: If the text content identified based on the second lip image information matches the text content corresponding to the second song, it is determined that the first user intends to sing.

11. The method according to any one of claims 8 to 10, characterized in that, The method also includes If no fourth voice message from the first user is detected within the second preset time period, or if the fourth voice message is detected within the second preset time period and does not match the text content or music waveform corresponding to the second song, the singing mode is turned off.

12. A control method, characterized in that, The method includes: Acquire first data collected by sensors in the cockpit, the first data including: image information of the first user, the first user's image information including: first lip image information and first body language information; If the first user makes a first action based on the first body language information, or if the first lip shape is identified based on the first lip image information, it is determined that the first user has the intention to sing. Start singing mode.

13. The method as described in claim 12, characterized in that, The first data also includes first voice information emitted by the first user, and the method further includes: The first song is determined based on the first lip image information and the first voice information; Control the sound-generating device to play the first song.

14. The method as described in claim 12 or 13, characterized in that, The method further includes: If no second voice message from the first user is detected within the first preset time period, or if the second voice message is detected within the first preset time period and does not match the text content or music waveform corresponding to the first song, the singing mode is turned off.

15. The method as described in claim 12, characterized in that, The first data also includes third voice information emitted by the first user. Prior to acquiring the first data collected by sensors within the cockpit, the method further includes: Control the sound-producing device to play the second song; Before determining that the first user intends to sing, the method further includes: The third voice information is determined to match the text content or music waveform corresponding to the second song.

16. The method as described in claim 15, characterized in that, Before determining that the first user intends to sing, the method further includes: It is determined that the text content corresponding to the first lip image information matches the text content corresponding to the second song.

17. The method as described in claim 15 or 16, characterized in that, The method further includes: If no fourth voice message from the first user is detected within the second preset time period, or if the fourth voice message is detected within the second preset time period and does not match the text content or music waveform corresponding to the second song, the singing mode is turned off.

18. The method according to any one of claims 12 to 17, characterized in that, The method further includes: The control prompt device indicates that the singing mode should be turned on.

19. The method according to any one of claims 12 to 18, characterized in that, The activation of the singing mode includes: activating the singing mode of the first area selected by the first user, or activating the singing mode of all areas of the cabin.

20. The method as described in claim 19, characterized in that, The method further includes: When a second user in the cabin has a do-not-disturb requirement, the singing mode in the second area of ​​the cabin should be kept off or turned off, and the second user is located in the second area.

21. The method as described in claim 19, characterized in that, The method of activating the singing mode includes: activating the singing mode in the first area; the method further includes: When it is determined from the second data that a third user in the cabin intends to sing, the singing mode is activated for all areas of the cabin or for the third area within the cabin, where the third user is located.

22. A control device, characterized in that, Includes modules or units for performing the method as described in any one of claims 1 to 11, or 12 to 21.

23. A control device, characterized in that, The device includes a processor and a memory, the processor being coupled to the memory, the memory being used to store computer programs or instructions, and the processor being used to execute the computer programs or instructions in the memory, such that the method of any one of claims 1 to 11 is performed.

24. A control device, characterized in that, The device includes a processor and a memory, the processor being coupled to the memory, the memory being used to store computer programs or instructions, and the processor being used to execute the computer programs or instructions in the memory, such that the method of any one of claims 12 to 21 is performed.

25. A chip, characterized in that, The chip includes circuitry for performing the method as described in any one of claims 1 to 21.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that, when executed on a computer, causes the computer to perform the method as described in any one of claims 1 to 21.

27. A computer program product, characterized in that, The computer product includes a computer program that, when run, causes the computer to perform the method as described in any one of claims 1 to 21.

28. A vehicle, characterized in that, Includes the apparatus as described in claim 23 or 24.

Images