Vehicle cabin linkage method and device, storage medium and program product

Abstract

The invention relates to a vehicle cabin linkage method and device, a storage medium and a program product. The method comprises the following steps: in response to target content detected in first multimedia content played in a vehicle cabin, determining whether a first emotion of a user is a matched target emotion; and under the condition that the first emotion is the matched target emotion, vehicle cabin linkage is triggered. Through the mode, the vehicle cabin linkage can be triggered according to the emotion feedback of the user on the refreshing scene, so that the linkage is prevented from being directly triggered in the refreshing scene. Therefore, according to the embodiment of the invention, personalized cabin linkage can be provided for the user, so that the use experience of the user is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of vehicle control, and more particularly, to a method, device, storage medium and program product for vehicle cabin linkage. BACKGROUND

[0002] With the development of technology, the entertainment attribute of a vehicle as a means of transportation is also increasingly valued. Based on the progress of multimedia systems, intelligent devices and the like in the vehicle cabin, the experience of a user when driving or riding in the vehicle is also increasingly rich. With the continuous improvement of user demand, the entertainment function of the vehicle is becoming more and more intelligent.

[0003] In the development of the entertainment function of the vehicle, personalized customization for the user is an important task to achieve the intelligentization of the entertainment function. For example, in the process in which the user performs activities such as navigation, music listening, video watching and game playing through various devices of the vehicle, how to provide personalized interaction for the user becomes an important direction for the development of the entertainment function of the vehicle. SUMMARY

[0004] In a first aspect of embodiments of the present disclosure, a method for vehicle cabin linkage is provided, including determining whether a first emotion of a user is a matching target emotion in response to detecting a target content in first multimedia content played in a vehicle cabin; and triggering vehicle cabin linkage in a case where the first emotion is the matching target emotion.

[0005] In a second aspect of embodiments of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a memory coupled to the at least one processor and having stored thereon instructions that, when executed by the at least one processor, cause the electronic device to perform actions including determining whether a first emotion of a user is a matching target emotion in response to detecting a target content in first multimedia content played in a vehicle cabin; and triggering vehicle cabin linkage in a case where the first emotion is the matching target emotion.

[0006] In a third aspect of embodiments of the present disclosure, a computer-readable storage medium having stored thereon a computer program is provided, the program being executed by a processor to implement a method for vehicle cabin linkage. The method includes determining whether a first emotion of a user is a matching target emotion in response to detecting a target content in first multimedia content played in a vehicle cabin; and triggering vehicle cabin linkage in a case where the first emotion is the matching target emotion.

[0007] In a fourth aspect of the embodiments of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer readable medium and includes machine executable instructions that, when executed, cause a machine to implement a method of vehicle cabin linkage. The method determines whether a first emotion of a user is a matching target emotion in response to detecting a target content in a first multimedia content played in a vehicle cabin; and triggers the vehicle cabin linkage in a case where the first emotion is the matching target emotion.

[0008] It should be understood that all statements herein made regarding the foregoing aspects of the embodiments of the present disclosure are intended to encompass both the specific and general features, and that the foregoing statements are not intended to limit the scope of the present disclosure. Other aspects of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other features, aspects and advantages of the present embodiments will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to like elements, wherein:

[0010] Figure 1 A schematic diagram illustrating an example environment in which some embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A flowchart illustrating a method of vehicle cabin linkage of some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrating a process of establishing a matching rule of some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram illustrating a user emotion matching rule of some embodiments of the present disclosure is shown;

[0014] Figure 5A A schematic diagram illustrating a process of vehicle cabin linkage of some embodiments of the present disclosure is shown;

[0015] Figure 5B A schematic diagram illustrating another process of vehicle cabin linkage of some embodiments of the present disclosure is shown;

[0016] Figure 6 A schematic diagram illustrating an apparatus of vehicle cabin linkage of some embodiments of the present disclosure is shown; and

[0017] Figure 7 A block diagram illustrating an apparatus that can implement various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather, these embodiments are provided so as to more completely and thoroughly understand the present disclosure. It is understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of embodiments of the present disclosure, the term "comprising" and its conjugations are to be understood to be open-ended, i.e., "including but not limited to". The term "based on" is to be understood as "based at least in part on". The term "one embodiment" or "an embodiment" is to be understood as "at least one embodiment". The term "a first" or "the first" is to be understood as "at least one" or "one or more". Other explicit or implicit definitions can also be included below.

[0020] As described above, in the development of the entertainment function of the vehicle, personalization for the user is an important task to realize the intelligentization of the entertainment function. The vehicle cabin has gradually evolved into a mobile multimedia entertainment space, in which the user can operate various application programs such as navigation, music, video, games, etc. through the vehicle machine system or the user mobile device to meet different user needs. In the cabin of some vehicles, there is a lighting system for creating atmosphere, such as atmosphere lights with adjustable color and brightness, etc. The control authority of the atmosphere lights is usually managed by the vehicle machine system, which can issue control instructions to adjust the color, brightness, etc. of the atmosphere lights according to the preset mode or the real-time received data signal.

[0021] In addition, some vehicles are also equipped with a fragrance system that can release different fragrances in the vehicle to create different atmosphere experiences, and the vehicle machine system can control the release time and intensity of the fragrance, etc. In addition, the vehicle machine system can also issue control instructions to systems that can interact with the user in a sensory way, such as the sound system, seat massage system, or air conditioning system of the vehicle, to change their working states. By centrally controlling these environmental hardware systems, the vehicle machine system can adjust the environment in the cabin according to the received external data to create an immersive experience for the user.

[0022] Relatedly, when the user is engaged in entertainment activities in the vehicle cabin, for example, when the user is playing games, watching a ball game, watching a movie, or karaoke, when the multimedia content such as the ball game or the movie is played to the climax scene, the sound, atmosphere light strip, seat, air conditioning, etc. in the cabin can be activated relying on the computing resources of the vehicle and the application software of the vehicle, so as to realize the linkage between the vehicle cabin and the user. However, since different users have different emotional feedbacks at the same climax scene, the way of directly linking the cabin at the climax scene in the related art lacks pertinence to the user.

[0023] To this end, a method for vehicle cabin linkage is provided in embodiments of the present disclosure. The method can detect whether there is a sweet spot content in the multimedia content while the multimedia content is being played, and if there is, it can further determine whether the user emotion is a sweet spot emotion. In the case where the user emotion is a sweet spot emotion, the vehicle cabin linkage device is triggered to link. In this way, the vehicle cabin linkage can be triggered in response to the user's emotional feedback to the sweet spot scene, thereby avoiding triggering the linkage directly at the sweet spot scene. Therefore, embodiments of the present disclosure can provide personalized cabin linkage for the user, thereby improving the user's experience.

[0024] Figure 1 A schematic diagram of an example environment 100 in which some embodiments of the present disclosure can be implemented is shown. Referring to Figure 1 , the example environment 100 includes a vehicle 102, which includes a control unit 104 and a vehicle cabin linkage device 110. The control unit 104 can be any device with computing or processing capability. For example, the control unit 104 can be an infotainment system, a desktop computer, a laptop computer, a tablet computer, a server, a mobile device, an electronic control unit of the vehicle 102 (including a vehicle control unit and control units of subsystems), etc. The control unit 104 can send control instructions to the vehicle cabin linkage device 110 to change the state of the vehicle cabin linkage device 110.

[0025] The vehicle cabin linkage device 110 is any component in the cabin of the vehicle 102 that can have a sensory interaction with the user. For example, the vehicle cabin linkage device 110 can include ambient lights, sound, air conditioning, seats, fragrance, etc. in the cabin. For example, the ambient lights can change color, change brightness, or flash, etc. in response to the user's emotion to the multimedia content played in the vehicle cabin, the sound can change the size of the sound, play a specific sound, etc. in response to the user's emotion to the multimedia content played in the vehicle cabin, the air conditioning can change the strength of the wind, etc. in response to the user's emotion to the multimedia content played in the vehicle cabin.

[0026] In embodiments provided by the present disclosure, the control unit 104 can detect whether there is a sweet spot content (which can be referred to as target content) in the multimedia content 106 (which can be referred to as first multimedia content) while the multimedia content 106 is being played in the vehicle cabin. In some embodiments, the multimedia content 106 includes, but is not limited to, video, song, game, audio reading, broadcast, etc.

[0027] Then, if the sweet spot content is detected, the control unit 104 can obtain the current user emotion 108 (may be referred to as a first emotion) of the user to the multimedia content 106, and determine whether the user emotion 108 is the sweet spot emotion (may be referred to as a target emotion). Then, in the case where the user emotion 108 is the sweet spot emotion, the control unit 104 can trigger the vehicle cabin linkage device 110 to perform linkage.

[0028] In this way, the vehicle cabin linkage can be triggered for the emotional feedback of the user to the sweet spot scene when the multimedia content 106 is played, so as to avoid triggering the vehicle cabin linkage device 110 to perform linkage directly at the sweet spot scene. Therefore, the embodiments of the present disclosure can provide personalized cabin linkage for the user, thereby improving the user experience.

[0029] It should be understood that the architecture and functions in the example environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure. Embodiments of the present disclosure can also be applied to other environments with different structures and / or functions.

[0030] The following will be described in combination with Figures 2 to 7 The process according to the embodiments of the present disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are all exemplary, and are not used to limit the protection scope of the present disclosure. It should be understood that the following described embodiments can also include additional actions not shown and / or can omit the shown actions, and the scope of the present disclosure is not limited in this respect.

[0031] Figure 2 A flowchart of a method 200 of vehicle cabin linkage of some embodiments of the present disclosure is shown. In some embodiments, the method 200 can be performed by the control unit 104 in the example environment 100 shown in FIG. 1. Figure 1 It should be understood that although the following is described with the control unit 104 as the execution subject, the method 200 can also be performed by other devices. The method 200 can also include additional actions not shown and / or can omit the shown actions, and the scope of the present disclosure is not limited in this respect.

[0032] At 202, if the target content is detected in the first multimedia content played in the vehicle cabin, it is determined whether the first emotion of the user is the matching target emotion. In some embodiments, when the multimedia content 106 is played in the vehicle cabin, the control unit 104 can detect in real time whether there is sweet spot content in the multimedia content 106. If there is sweet spot content, the control unit 104 can further determine whether the user emotion 108 is the sweet spot emotion.

[0033] At 204, in a case where the first emotion is a matching target emotion, a vehicle cabin linkage is triggered. In some embodiments, when it is determined that the user emotion 108 is a peak emotion, the control unit 104 can control the vehicle cabin linkage device 110 to perform linkage. For example, when the user is in a peak emotion such as excitement, the control unit 104 can control the vehicle cabin linkage device 110 to provide corresponding linkage feedback to the user.

[0034] In this way, the vehicle cabin linkage can be triggered in response to the user's emotional feedback to the peak scene, thereby avoiding triggering the linkage directly at the peak scene. Therefore, the embodiments of the present disclosure can provide personalized cabin linkage for the user, thereby improving the user's use experience.

[0035] In some embodiments, to increase the interactivity between the vehicle cabin and the user, the user image can be captured by a camera, and the user emotion can be directly recognized by a computer vision-based emotion recognition model, and then it is determined whether the user is in a peak emotion. However, due to the camera accuracy, occlusion, non-obvious facial expression (the user's inner fluctuation is large), and dark light conditions (especially the light conditions in the cabin are often poor in the case of atmosphere lamp linkage), the accuracy of peak emotion recognition is often low. Moreover, the above-mentioned method can only recognize and perform cabin linkage after the user produces a peak emotion (i.e., the peak emotion recognition has a lag), so that the timing of the cabin linkage is delayed compared to the timing of the user producing the peak emotion, thereby missing the best timing of the cabin linkage.

[0036] Therefore, in some embodiments, the control unit 104 can pre-establish a user emotion matching rule (which can be referred to as a matching strategy) between the peak content (which can be referred to as target content) of the multimedia content (which can be referred to as second multimedia content) and the user's peak emotion. When playing the multimedia content 106 (which can be referred to as first multimedia content. It should be understood that in the present disclosure, the first multimedia content is the currently played content, and the second multimedia content is the content played when the matching strategy is constructed, i.e., the historical content), the control unit 104 can obtain the pre-established user emotion matching rule. Then, the control unit 104 can detect the peak content in the multimedia content 106, and then determine whether the user emotion 108 is a peak emotion based on the detected peak content in the multimedia content 106 and the user emotion matching rule. In a case where the user emotion 108 is a peak emotion, the control unit 104 can trigger the vehicle cabin linkage device 110 to perform linkage. In some embodiments, the user emotion matching rule can also include peak content obtained by other means and the user's emotional feedback to the peak content.

[0037] In this way, the control unit 104 can directly determine whether the user emotion 108 is a wow emotion based on the detected wow content in the multimedia content 106, without the need for identification by the computer vision emotion recognition model, thereby avoiding the influence of the user facial expression being not obvious and the dark light condition and the like on the identification result. Moreover, through the wow content of the multimedia content 106 and the user emotion matching rule, the wow emotion of the user can be pre-judged, avoiding the situation that the identification and the cabin linkage can only be performed after the user produces the wow emotion, thereby improving the accuracy of the wow emotion identification and the accuracy of the cabin linkage, and further improving the user's use experience.

[0038] In some embodiments, in the process of establishing the user emotion matching rule, the control unit 104 obtains the user features (which can be referred to as second features) of the user in the process of playing the multimedia content (which can be referred to as second multimedia content) in the vehicle cabin, and determines the user emotion (which can be referred to as second emotion) based on the user features. Then, the control unit 104 determines whether the user emotion is a wow emotion, and obtains the wow content in the multimedia content that triggers the wow emotion of the user when the user emotion is a wow emotion. Further, the control unit 104 constructs the user emotion matching rule according to the wow content obtained from the multimedia content and the wow emotion of the user. In some embodiments, the user features can include facial features, voice features, physical features, and interaction features of the multimedia content, and the like. The above process will be described in detail below in combination with Figures 3-5B .

[0039] Figure 3 A schematic diagram of a process 300 of establishing a matching rule of some embodiments of the present disclosure is shown. In some embodiments, the process 300 can be performed by the control unit 104 in the example environment 100 shown in Figure 1 . It should be understood that although the following is described with the control unit 104 as the execution subject, the process 300 can also be performed by other devices. The process 300 can also include additional actions not shown and / or can omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0040] In some embodiments, the sensor 302 includes vehicle cameras (including infrared cameras and RGB cameras, etc.), microphones, electronic screens, etc. The control unit 104 can acquire an image of the user through the camera, and then perform face detection 304 according to the image of the user. In some embodiments, when playing a multimedia stream 316 (which can be referred to as second multimedia content) in the vehicle cabin, the control unit 104 receives the acquired image of the user from the camera, and detects the image of the user to extract a face region from the image. Then, the control unit 104 extracts a key point feature of the face (which can be used as a facial feature) based on the face region, such as an eye corner key point feature, etc. Then, the control unit 104 performs user emotion recognition 312 by classifying the key point feature to determine the user emotion.

[0041] In some embodiments, the image of the user can be acquired through the vehicle camera, and then the control unit 104 can perform vital sign detection 306 according to the image of the user. In some embodiments, when playing a multimedia stream 316 in the vehicle cabin, the control unit 104 receives the acquired image of the user from the camera, extracts a region of interest in the image and calculates a region feature, and then classifies the region feature to obtain vital sign features of the user, such as heart rate, blood oxygen saturation, blood pressure, respiratory rate, and pulse, etc. Further, the control unit 104 performs user emotion recognition 312 to determine the user emotion according to the vital sign features.

[0042] For example, the control unit 104 can detect a region of interest in each frame, such as a face or a forehead, etc., perform feature extraction on the region of interest, thereby obtaining periodic features and mean values of each component signal of the RGB signal, filter the sequence trend part in the RGB signal through filtering, retain the signal fluctuation information, and perform Fourier transform on the pulsation information to obtain a signal spectrum, which is used as the region feature of the region of interest. Alternatively, after determining the region of interest, the region of interest is spatially divided to obtain a plurality of sub-regions, and a spatiotemporal mapping diagram in the plurality of sub-regions in each frame is acquired as the region feature of the region of interest.

[0043] In some embodiments, the user sound can be collected by a vehicle microphone, and then the control unit 104 can perform voice detection 308 according to the user sound. In some embodiments, when the multimedia stream 316 is played in the vehicle cabin, the control unit 104 receives the sound in the vehicle cabin from the microphone, and cancels the sound of the multimedia stream 316 and the noise during driving in the sound in the vehicle cabin, to obtain the user sound after cancellation. Then, the control unit 104 determines the sound characteristics such as the type, decibel, energy, and spectrum of the user sound. Further, then, the control unit 104 performs user emotion recognition 312 to determine the user emotion according to the sound characteristics such as the type, decibel, energy, and spectrum. In some embodiments, the control unit 104 can perform echo cancellation on the sound such as music, movie, etc. played by the vehicle speaker, so as to only obtain the sound emitted by the user, including the sound of user breathing, speaking, singing, etc., to identify the type through the sound event, and to perform decibel, energy, spectrum centroid, etc. detection, so as to comprehensively measure the intensity of the user sound.

[0044] In some embodiments, the user click event stream (which can be referred to as click information) can be collected by an electronic screen, and then the control unit 104 can perform click detection 310 according to the user click event stream. In some embodiments, when the multimedia stream 316 is played in the vehicle cabin, the control unit 104 obtains the click event stream of the user on the play interface of the multimedia stream 316 from the screen, and determines the interaction characteristics such as the force and frequency of the user clicking the play interface. Then, the control unit 104 performs user emotion recognition 312 to determine the user emotion according to the interaction characteristics such as the force and frequency of the user clicking the play interface. In some embodiments, the play interface can be the screen of the vehicle multimedia system, or the screen of a mobile terminal (such as a mobile phone) connected with the vehicle.

[0045] It should be understood that in the user emotion recognition 312 performed by the control unit 104, the user emotion can be determined based on one of the face detection 304, the vital sign detection 306, the voice detection 308, and the click detection 310, or the user emotion can be determined based on multiple detection results, for example, each detection result can be weighted to determine the user emotion. In this way, the accuracy and flexibility of determining the user emotion can be improved, and more vehicle scenarios can be adapted to. In some embodiments, after the control unit 104 determines the user emotion, the control unit 104 can further perform a highlight emotion recognition 314 to determine whether the user emotion is a highlight emotion. For example, whether the user emotion is a highlight emotion can be determined according to the emotion type, duration, emotion degree, etc. of the user emotion.

[0046] In some embodiments, after the control unit 104 determines that the user emotion is a high point emotion, the multimedia content corresponding to each high point emotion can be cut in the multimedia stream 316. For example, the time sequence of each high point emotion in the multimedia stream 316 can be determined, and the corresponding multimedia content in the multimedia stream 316 can be cut according to the time sequence. Then, the control unit 104 performs high point scene labeling 318 on the cut multimedia content, thereby labeling the high point content in the high point scene. Further, the control unit 104 establishes a user emotion matching rule 320.

[0047] In some embodiments, in the process of establishing the user emotion matching rule 320, the control unit 104 establishes a decision tree according to the labeled high point content and the user's high point emotion. In some embodiments, the decision tree has a plurality of hierarchical classification rules of a logical system, and the classification rule at the end of the logical system is used to divide whether the user emotion is a high point emotion. In this way, whether the user emotion is a high point emotion can be directly determined based on a plurality of classification rules, thereby improving the efficiency of determining the user emotion.

[0048] In some embodiments, the high point content includes at least one of audio information, image information, and video information of the multimedia content, the audio information includes at least one of prompt sound, sound effect, music type, and audio transition, the image information includes at least one of image semantics, prompt word, color distribution, and image effect, and the video information includes at least one of video transition and video style. It should be understood that the above content is only a listing of the high point content in the present disclosure, and does not limit the high point content.

[0049] Figure 4 A schematic diagram of the user emotion matching rule 320 of some embodiments of the present disclosure is shown. In some embodiments, the user emotion matching rule 320 is a decision tree model. In the user emotion matching rule 320, the root node 402, the internal node 404, the internal node 406, and the leaf node 408, the leaf node 410, the leaf node 412, the leaf node 414, and the leaf node 416 are included. The root node 402, the internal node 404, and the internal node 406 are used to make decisions based on classification rules, and the leaf node 408, the leaf node 410, the leaf node 412, the leaf node 414, and the leaf node 416 are the decision results.

[0050] In the root node 402, the classification rule is based on the game sound. If the game sound is settlement (e.g., the settlement of benefits at the end of the game), the leaf node 408 is obtained as the decision result of the awesome point emotion, and if the game sound is dialogue (e.g., the dialogue of the game character), the leaf node 410 is obtained as the decision result of the general emotion. If the game sound is a prompt, enter the internal node 404, and the classification rule is based on the prompt type. If the prompt type is operation (e.g., prompting the user to purchase game equipment), the leaf node 412 is obtained as the decision result of the general emotion, and if the prompt type is combat (e.g., the battle between two and multiple game characters), enter the internal node 406, and the classification rule is based on the combat prompt. If the combat prompt is victory, the leaf node 414 is obtained as the decision result of the awesome point emotion, and if the combat prompt is failure, the leaf node 416 is obtained as the decision result of the general emotion.

[0051] Figure 5A A schematic diagram of a process 500A of vehicle cabin linkage of some embodiments of the present disclosure is shown. In some embodiments, the process 500A can be performed by the control unit 104 in the example environment 100 shown in FIG. 1. It should be understood that although the following is described with the control unit 104 as the performing subject, the process 500A can also be performed by other devices. The process 500A can also include additional actions not shown and / or can omit the actions shown, and the scope of the present disclosure is not limited in this regard. Figure 1

[0052] In some embodiments, after the user emotion matching rule 320 is established, the control unit 104 can acquire the current multimedia content 106 and detect whether there is awesome point content in the multimedia content 106. In the case where there is awesome point content, the control unit 104 performs awesome point emotion matching 501 according to the user emotion matching rule 320 and the awesome point content to determine whether the user emotion 108 is an awesome point emotion. Specifically, the control unit 104 can classify the awesome point content of the multimedia content 106 according to the multiple classification rules in the awesome point emotion matching rule 320, so as to determine whether the user emotion 108 is an awesome point emotion 504 when the awesome point content. Further, the control unit 104 performs vehicle cabin linkage 506. In this way, it can be quickly determined whether the user emotion is an awesome point emotion, and the accuracy of the determination of the awesome point emotion is improved.

[0053] ​In some embodiments, the sweet spot content can be both image sweet spot content and audio sweet spot content in the multimedia content 106. The control unit 104 can extract images and audio in the multimedia content 106 and determine whether there are corresponding image sweet spot content and audio sweet spot content in the images and audio. In some embodiments, the multimedia content 106 can be a game, the image sweet spot content can be a text prompt or a pattern prompt displayed on the game interface when the user reaches a specified condition in the game, and the audio sweet spot content can be a voice prompt or a music prompt played when the user reaches the specified condition in the game. For example, when the user defeats three enemy units in succession in the game, the “three hits in a row” text displayed on the game interface and the “three hits in a row” voice played can be the image sweet spot content and the audio sweet spot content, respectively.

[0054] In some embodiments, during the process in which the control unit 104 performs the vehicle cabin linkage 506, the following cabin devices in the vehicle can be triggered to link, including ambient lights, interior cabin lights, vehicle audio, seats, air conditioners, or fragrances, etc. For example, when the user emotion is a sweet spot emotion, the seats can be controlled to vibrate and the ambient lights can be controlled to flash. It should be understood that the above enumeration is only an example of the vehicle cabin linkage devices 110.

[0055] Figure 5B A schematic diagram of another process 500B of vehicle cabin linkage of some embodiments of the present disclosure is shown. In some embodiments, the process 500B can be performed by the control unit 104 in the example environment 100 shown in FIG. 1. It should be understood that although the following is described with the control unit 104 as the performing subject, the process 500B can also be performed by other devices. The process 500B can also include additional actions not shown and / or can omit the actions shown, and the scope of the present disclosure is not limited in this respect. Figure 1

[0056] In some embodiments, the control unit 104 performs face detection 508, vital sign detection 510, voice detection 512, and click detection 514 based on the sensors 320, and then performs user emotion recognition 516 and sweet spot emotion recognition 518 based on the multiple detection results, so as to obtain a first result that the user emotion 108 at this time is a sweet spot emotion. The specific process of determining this first result in the present embodiment is the same as the user emotion recognition 312 and the sweet spot emotion recognition 314 in the process 300, and thus will not be described in detail.

[0057] ​Then, the control unit 104 can obtain the current multimedia content 106, perform the highlight emotion matching 502 according to the first result, the user emotion matching rule 320 and the multimedia content 106, specifically, the control unit 104 classifies the user features of the multimedia content 106 according to the plurality of classification rules in the highlight emotion matching rule 320, thereby obtaining the second result that the user emotion 108 at this time is the highlight emotion. After obtaining the two results, the control unit 104 can determine that the user emotion 108 is the highlight emotion 504, and then the control unit 104 performs the vehicle cabin linkage 506. In this way, the control unit 104 can combine the real-time detection result of the user emotion, and perform secondary verification on the emotion recognition result matched based on the user emotion matching rule 320, thereby further improving the accuracy of highlight emotion recognition.

[0058] In some embodiments, after obtaining the first result that the user emotion 108 is the highlight emotion, the classification rules of the multimedia content 106 can be further extended, thereby obtaining classification rules different from the classification rules already existing in the user emotion matching rule 320. Then, based on the classification rules different from the classification rules already existing in the user emotion matching rule 320, the classification rules already existing in the user emotion matching rule 320 are updated. In this way, the accuracy and timeliness of the user emotion matching rule 320 can be improved, thereby improving the accuracy of user emotion recognition and user experience.

[0059] Figure 6 A schematic diagram of an apparatus 600 for vehicle cabin linkage of some embodiments of the present disclosure is shown. As shown, the apparatus 600 includes a target emotion determination unit 602 configured to determine whether a first emotion of a user is a matching target emotion in response to detecting target content in a first multimedia content played in a vehicle cabin. The apparatus 600 also includes a vehicle cabin linkage unit 604 configured to trigger a vehicle cabin linkage in a case where the first emotion is the matching target emotion. Figure 6

[0060] It can be understood that, by using the apparatus 600 of the present disclosure, at least one of the many advantages that can be achieved by the method or process described above can be achieved. For example, whether there is highlight content in the multimedia content can be detected while the multimedia content is played, and if there is, it can be further determined whether the user emotion is a highlight emotion. In a case where the user emotion is a highlight emotion, the vehicle cabin linkage device is triggered to link. In this way, the vehicle cabin linkage can be triggered in response to the emotional feedback of the user to the highlight scene, thereby avoiding triggering the linkage directly at the highlight scene. Therefore, the embodiments of the present disclosure can provide personalized cabin linkage for the user, thereby improving the user's use experience.

[0061] ​Figure 7 A schematic structural diagram of an example device 700 that can be used to implement some embodiments according to the present disclosure is shown. The device 700 can be implemented as Figure 1 Embodiments of the present disclosure do not limit the specific type of device 700, which can be implemented as a control unit 104 in FIG. 1, etc. As shown in FIG. 7, the device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for operation of the device 700 can also be stored in the RAM 703. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Figure 7

[0062] Various components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; the storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0063] The processing unit 701 can perform various methods and / or processes described above, such as the method shown in FIG. 6. For example, in some embodiments, the method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU 701, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the CPU 701 can be configured to perform the method by any other appropriate means, such as by means of firmware, etc. Figure 2

[0064] The functions described above in this detailed description can be performed at least in part by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0065] ​​In some embodiments, the methods and processes described above can be tied to a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions thereon for purposes of causing a processor to implement various aspects of the present disclosure.

[0066] Program code used by or in connection with the described embodiments can be written in any combination of one or more programming languages, including an object oriented programming language and conventional procedural programming languages. The program code can execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0067] The computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0068] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0069] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD), or other disk storage, or any combination of these. The computer readable program instructions can also be downloaded to a computer, other programmable data processing apparatus, or other device from a computer readable storage medium or to an appropriate computer or other programmable data processing apparatus, including a RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD), or other disk storage, or any combination of these. Also, computer readable program instructions can be transmitted in a computer readable transmission medium. A computer readable transmission medium includes a single network link, or a plurality of network links. The computer readable transmission medium can include a computer readable signal directed, for example, to an appropriate computer or other programmable data processing apparatus over a communication network. These computer readable program instructions might be implemented also to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0070] In the information of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0071] The computer program product of the second aspect can include a computer readable storage medium. The computer readable storage medium can include instructions. The instructions can include one or both of: instructions for causing a computer to implement a method as described above; and instructions for causing a computer to operate based on a computer program as described above.

[0072] Further, although operations are depicted in a particular, chronological sequence, this should not be understood as requiring such order unless specifically specified that an operation is order dependent. One of ordinary skill in the art will recognize that state of the art operations can be performed in many ways, and that the illustrations and descriptions of operations should not be construed as requiring a particular, chronological sequence, unless otherwise specifically stated. Also, where specific implementations are discussed, it should be understood that the same can apply to other implementations. For example, where a particular implementation is discussed, it should be understood that the same can apply to any of the implementations described herein. Similarly, where a particular feature is discussed, it should be understood that the same can apply to any of the features described herein.

[0073] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0074] Various modifications and changes can be made as would be obvious to a person of ordinary skill in the art having the benefit of this disclosure without departing from the scope and spirit of the description or the scope of the claims. It is intended that the scope of the claims include all such modifications and changes. The description and the claims should not be construed as limited to the examples described herein. The various embodiments described herein can be implemented in a computer program product, which can include a computer-readable medium bearing computer program code, machine-executable instructions, or data structures storing instructions for use by an instruction execution system, processor, or computer. The computer-readable medium can be manufactured as single integrated circuits, or over multiple, distributed integrated circuits. The computer-readable medium can be manufactured as a single program product or as multiple program products.

Claims

1. A method for vehicle cockpit linkage, comprising: In response to detecting target content in first multimedia content played in the vehicle cabin, determine whether the user’s first emotion is a matching target emotion; as well as If the first emotion is the target emotion that is matched, the vehicle cabin linkage is triggered.

2. The method according to claim 1, wherein the target content includes image content and / or audio content, and the method further includes: Acquire images and / or audio from the first multimedia content; as well as Determine whether the image content exists in the image, and / or determine whether the audio content exists in the audio.

3. The method according to claim 2, wherein the first multimedia content includes a game, the image content includes text prompts and / or graphic prompts related to the user achieving specified conditions in the game, and the audio content includes voice prompts and / or music prompts related to the user achieving the specified conditions in the game.

4. The method according to any one of claims 1 to 3, wherein triggering the vehicle cabin linkage when the first emotion is the matched target emotion includes: If the first emotion is the matched target emotion, trigger at least one of the following to activate the linkage: ambient lighting, cabin lighting, vehicle audio, seats, air conditioning, or fragrance.

5. The method of claim 1, wherein determining whether the user's first emotion is a matching target emotion comprises: Obtain a matching strategy, which is related to the user's target emotion and the target content in the second multimedia content; as well as Based on the target content in the first multimedia content and the matching strategy, it is determined whether the first emotion is the target emotion to be matched.

6. The method according to claim 5, wherein determining whether the first emotion is the matched target emotion based on the target content in the first multimedia content and the matching strategy includes: Based on the target content in the first multimedia content and the matching strategy, determine whether the first emotion is the first result of the matching target emotion; Based on the user's first characteristic, a second result is determined as to whether the first emotion is the matched target emotion; as well as Based on the first result and the second result, determine whether the first emotion is the matched target emotion.

7. The method according to claim 5, wherein the process of establishing the matching strategy includes: Based on the user's second characteristic, determine the user's second emotion towards the second multimedia content; If the second emotion is the target emotion, determine the target content in the second multimedia content; as well as The matching strategy is established based on the target emotion and the target content in the second multimedia content.

8. The method of claim 7, wherein the second feature includes facial key point features, and determining the user's second emotion toward the second multimedia content based on the user's second feature comprises: While playing the second multimedia content, the user's image is acquired via the vehicle's camera; Based on the facial region in the image, the key point features are determined; as well as Based on the key point features, the user's second emotion towards the second multimedia content is determined.

9. The method of claim 7, wherein the second feature includes vital signs, the vital signs including at least one of heart rate, blood oxygen saturation, blood pressure, respiratory rate, and pulse, and determining the user's second emotion toward the second multimedia content based on the user's second feature includes: While playing the second multimedia content, the user's image is acquired via the vehicle's camera; The vital signs are determined based on the regional features of the region of interest in the image; as well as Based on the aforementioned physical characteristics, the user's second emotion towards the second multimedia content is determined.

10. The method of claim 7, wherein the second feature includes a sound feature, the sound feature including at least one of type, decibel, energy, and spectrum, and determining the user's second emotion toward the second multimedia content based on the user's second feature includes: While playing the second multimedia content, the sound inside the vehicle cabin is acquired via the vehicle microphone; Filter out sounds and / or driving noises related to the second multimedia content from the sound; Based on the filtered sound, the sound features are determined; as well as Based on the sound characteristics, the user's second emotion towards the second multimedia content is determined.

11. The method of claim 7, wherein the second feature includes an interaction feature, the interaction feature including the intensity and / or frequency of the click, and determining the user's second emotion towards the second multimedia content based on the user's second feature includes: When playing the second multimedia content, obtain the user's click information on the playback interface of the second multimedia content, the playback interface including the screen in the vehicle cabin and / or the screen of the mobile terminal connected to the vehicle cabin; Based on the click information, the interaction features are determined; as well as Based on the interaction features, the user's second emotion towards the second multimedia content is determined.

12. The method according to any one of claims 5 to 11, wherein the matching strategy is established based on the structure of a decision tree.

13. An electronic device, comprising: At least one processor; as well as A memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the electronic device to perform the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 12.

15. A computer program product tangibly stored on a non-volatile computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 12.

Images