Out-vehicle man-vehicle interaction method and device, vehicle and storage medium

Abstract

The invention discloses an outside-vehicle man-vehicle interaction method and device, a vehicle and a storage medium. The method comprises the following steps: acquiring voice information of a target user outside a vehicle; determining a voice understanding result corresponding to the voice information, an emotion analysis result corresponding to the voice information and dialogue content for replying the voice information through a voice processing model; determining a target feedback mode corresponding to the voice information based on the voice understanding result, the sentiment analysis result and the dialogue content; and based on the target feedback mode, controlling external equipment of the vehicle to execute multi-mode feedback, wherein the multi-mode feedback at least comprises playing of the dialogue content. According to the scheme, through accurate language understanding, emotion perception and multi-mode feedback including dialogue reply, natural, intimate and rich out-of-vehicle human-vehicle interaction experience is provided, the multi-mode feedback mode can be adjusted according to the emotional state of the target user through interaction by adding emotion perception, and more personalized interaction service is achieved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of intelligent vehicle technology, and in particular, to a method, device, vehicle and storage medium for vehicle-to-vehicle interaction. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, human-vehicle interaction has become increasingly intelligent. Human-vehicle interaction is the core link in the development of smart cars, and can have a variety of interactive forms, such as voice interaction, tactile interaction, auditory interaction, etc., providing users with unprecedented convenience and comfort.

[0003] In the existing technology, it is possible to control the in-vehicle equipment through voice commands outside the vehicle. Its actual purpose is to control the in-vehicle environment, and more attention is paid to the execution of functional commands, such as controlling the vehicle through voice commands to play music, turn on the in-vehicle air conditioner, etc. The interaction method is relatively simple. Summary of the invention

[0004] The present invention provides a method, device, vehicle and storage medium for human-vehicle interaction outside a vehicle, which can enrich the interactive mode of human-vehicle interaction outside a vehicle and enhance the interactive experience.

[0005] In a first aspect, an embodiment of the present invention provides a vehicle-vehicle interaction method outside a vehicle, comprising:

[0006] Acquire voice information of a target user outside the vehicle;

[0007] Determine, through a speech processing model, a speech understanding result corresponding to the speech information, a sentiment analysis result corresponding to the speech information, and a conversation content in response to the speech information;

[0008] Determining a target feedback mode corresponding to the speech information based on the speech understanding result, the sentiment analysis result and the conversation content;

[0009] Based on the target feedback mode, the vehicle external device is controlled to perform multimodal feedback, and the multimodal feedback at least includes playing the conversation content.

[0010] In a second aspect, an embodiment of the present invention provides a vehicle-person interaction device outside a vehicle, including:

[0011] An acquisition module, used for acquiring voice information of a target user outside the vehicle;

[0012] A speech processing module, used to determine the speech understanding result corresponding to the speech information, the sentiment analysis result corresponding to the speech information, and the dialogue content of replying to the speech information through a speech processing model;

[0013] A feedback mode determination module, used to determine a target feedback mode corresponding to the speech information based on the speech understanding result, the sentiment analysis result and the conversation content;

[0014] A feedback execution module is used to control the vehicle external device to execute multimodal feedback based on the target feedback mode, and the multimodal feedback at least includes the playback of the dialogue content.

[0015] In a third aspect, an embodiment of the present invention provides a vehicle, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to the first aspect.

[0019] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0020] The technical solution of the embodiment of the present invention processes the voice information of the target user through a voice processing model, determines the voice understanding result corresponding to the voice information, the emotion analysis result corresponding to the voice information, and the dialogue content of the reply voice information, determines the target feedback mode based on the determined content, and further controls the vehicle external device to perform multimodal feedback through the target feedback mode. This solution provides a natural, friendly and rich vehicle-to-vehicle interaction experience through accurate language understanding, emotion perception and multimodal feedback including dialogue replies. The addition of emotion perception enables the multimodal feedback method to be adjusted according to the emotional state of the target user during interaction, realizing more personalized interactive services.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a vehicle-outside-vehicle interaction method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 is a flow chart of a vehicle-outside-vehicle interaction method provided according to Embodiment 2 of the present invention;

[0025] Figure 3 is a schematic diagram of the structure of a vehicle-person interaction device outside a vehicle provided according to a third embodiment of the present invention;

[0026] Figure 4 Schematic diagram of the structure of a vehicle implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] Figure 1 It is a flow chart of a method for human-vehicle interaction outside a vehicle provided according to the first embodiment of the present invention. This embodiment can be applied to the situation of realizing human-vehicle interaction outside a vehicle. The method can be executed by a human-vehicle interaction device outside a vehicle. The device can be implemented in the form of software and / or hardware and integrated in the vehicle.

[0031] like Figure 1 As shown, the method includes:

[0032] S110: Acquire voice information of a target user outside the vehicle.

[0033] The target user may be a user interacting with the vehicle outside the vehicle. For example, the target user can interact with the vehicle outside the vehicle within a set distance range from the vehicle. The set distance range is not limited, for example, it can be a distance range within which the vehicle can obtain the voice information of the target user.

[0034] The target user's voice information may be information indicating the target user's voice. The specific content of the voice information is not limited, and may be content for information query and acquisition, task execution and management, entertainment interaction, emotional support, technical support, etc.

[0035] In this step, multiple high-sensitivity microphones can be set at different locations outside the vehicle to form a microphone array, and the microphone array can be used to obtain the voice information of the target user from different locations outside the vehicle. Optionally, the acquired voice information can suppress environmental noise by implementing a noise processing algorithm to obtain clear and accurate voice information.

[0036] S120. Determine, through a speech processing model, a speech understanding result corresponding to the speech information, a sentiment analysis result corresponding to the speech information, and a conversation content in response to the speech information.

[0037] The speech processing model may be a model capable of implementing speech processing, such as a Large Language Model (LLM), which is not limited here. The speech understanding result may be the result of speech recognition and natural language understanding of the speech information. The sentiment analysis result may be the result of sentiment analysis of the speech information. The conversation content may be the content used to reply to the speech information to realize the conversation.

[0038] In this step, the speech processing model can be used to perform speech recognition and natural language understanding on the speech information, identify the keywords and sentence structure in the speech information, and thus understand the needs of the target user and obtain the speech understanding result; the speech processing model can be used to perform sentiment analysis on the speech information, identify the emotional color of the target user in the speech information, such as calmness, joy, anxiety, sadness, anger, etc., and obtain the sentiment analysis result; the speech processing model can be used to generate the dialogue content of the reply speech information based on the understanding and sentiment analysis of the speech information, and the generated dialogue content can be strongly related to the user's needs and emotions. Among them, the dialogue content can be a sentence with emotional color, such as a sentence to soothe the target user's emotions; it can also be a sentence that provides the target user with an answer to the queried information; it can also be a sentence that provides entertainment interaction for the target user; it is not limited here.

[0039] S130: Determine a target feedback mode corresponding to the voice information based on the voice understanding result, the sentiment analysis result and the conversation content.

[0040] The target feedback mode may be a mode for providing feedback on voice information. The target feedback mode may be a feedback mode determined from a plurality of preset feedback modes. The preset feedback mode may be a feedback mode pre-set according to actual application needs, and each preset feedback mode may be a form of multi-modal feedback, such as a combination of multiple feedback forms such as voice response, visual feedback, and tactile feedback outside the vehicle.

[0041] In this step, the speech understanding results, sentiment analysis results and conversation content can be comprehensively considered, and among multiple preset feedback modes, the feedback mode with the greatest relevance to the target user's needs, the target user's emotional color in the speech information and the conversation with the target user is matched, that is, the target feedback mode corresponding to the speech information is determined.

[0042] S140: Controlling the vehicle external device to perform multimodal feedback based on the target feedback mode, wherein the multimodal feedback at least includes playing the conversation content.

[0043] In the embodiment of the present invention, the target feedback mode can be used to determine various forms of feedback voice information, such as voice response, visual feedback, tactile feedback, etc. It can be determined which vehicle external device is specifically responsible for each feedback form, and the operation that each vehicle external device should perform under the corresponding feedback form. The vehicle external device can be a device deployed outside the vehicle, which is not limited here.

[0044] Therefore, in this step, based on the target feedback mode, corresponding instructions can be transmitted to each vehicle external device that needs to perform multimodal feedback, so that the vehicle external devices that receive the instructions simultaneously execute the operations corresponding to the received instructions, thereby realizing multimodal feedback of voice information.

[0045] In one embodiment, the multimodal feedback further includes at least one of the following:

[0046] Changes in the color and flashing pattern of the vehicle's exterior lights; changes in the opening and closing states of the windows; changes in the action of the windshield wipers; changes in the position of the vehicle's door handles; and the playing of sound effects from the vehicle's exterior speakers.

[0047] Wherein, the vehicle external equipment at least includes the vehicle external lights, the vehicle windows, the wipers, the vehicle door handles, and the vehicle external speakers.

[0048] Changes in the color and flashing mode of the vehicle's exterior lights: The vehicle's exterior lights can have different colors, such as red, white, etc.; the flashing mode of the vehicle's exterior lights can be changed by changing the flashing frequency and flashing modes, such as continuous flashing, flashing at certain intervals, etc.

[0049] Changes in the switch status of the windows: Each window can have different opening degrees, such as fully open, half open, etc.; different windows can choose the same or different opening degrees.

[0050] Changes in wiper action: The wipers can have different wiping frequencies, and the wiper action can be changed by running or not running the wipers.

[0051] Changing the position of a vehicle door handle: If the vehicle door handle is a hidden door handle, the position of the vehicle door handle can be changed by extending or retracting the hidden door handle.

[0052] Playing of sound effects by external speakers: External speakers can play a variety of preset sound effects, which are not limited here.

[0053] Exemplarily, controlling the vehicle's external devices to perform multimodal feedback based on the target feedback mode can be implemented as follows: based on the target feedback mode, controlling the vehicle's external speakers to play the conversation content of the reply voice message, controlling the vehicle's external lights to flash red, and controlling the hidden door handles to continuously switch between extended and retracted states. The above three controls can be executed simultaneously.

[0054] Optionally, multimodal feedback can also be implemented by providing feedback notifications on the target user's mobile phone, combining various methods such as vibration, sound and image; it can also be implemented by providing real-time feedback to the target user through an in-car display combined with external sensors, which is not limited here.

[0055] The technical solution of the embodiment of the present invention processes the voice information of the target user through a voice processing model, determines the voice understanding result corresponding to the voice information, the emotion analysis result corresponding to the voice information, and the dialogue content of the reply voice information, determines the target feedback mode based on the determined content, and further controls the vehicle external device to perform multimodal feedback through the target feedback mode. This solution provides a natural, friendly and rich vehicle-to-vehicle interaction experience through accurate language understanding, emotion perception and multimodal feedback including dialogue replies. The addition of emotion perception enables the multimodal feedback method to be adjusted according to the emotional state of the target user during interaction, realizing more personalized interactive services.

[0056] Embodiment 2

[0057] Figure 2 is a flow chart of a vehicle-vehicle interaction method provided according to the second embodiment of the present invention. This embodiment is a further refinement based on the above-mentioned first embodiment. Figure 2 As shown, the method includes:

[0058] S201 , in response to an interactive wake-up operation of a target user outside a vehicle, waking up a human-vehicle interaction outside the vehicle.

[0059] The interactive wake-up operation may be an operation for waking up a person-vehicle interaction outside the vehicle. The interactive wake-up operation may be an operation in which the target user clicks a wake-up button, which may be a physical button provided outside the vehicle for implementing the interactive wake-up operation. The interactive wake-up operation may also be an operation in which the target user issues a voice containing a wake-up word, which may be a word for implementing the interactive wake-up operation, which is not limited here.

[0060] In this step, the human-vehicle interaction outside the vehicle can be awakened in response to the target user clicking the wake-up button, or when it is determined that the voice uttered by the target user contains a wake-up word.

[0061] S202: Verify the interaction authority of the target user and feed back the verification result.

[0062] When waking up the human-vehicle interaction outside the vehicle, the target user's interaction rights can be verified and the verification result can be fed back. The verification result can be a success or failure. If the verification is successful, the interaction can continue, and if the verification fails, the interaction stops. The verification result can be fed back in the form of a prompt tone played by the external speaker of the vehicle, which is not limited here.

[0063] In one embodiment, the target user's interactive authority verification method includes at least one of the following: biometric verification, smart key authority verification, and Bluetooth pairing verification.

[0064] Among them, biometric verification can be specifically implemented as facial recognition through a camera outside the vehicle, or voice recognition by collecting the target user's voice through a microphone outside the vehicle, or fingerprint recognition through a fingerprint recognition device outside the vehicle, which is not limited here. Smart key authority verification can be specifically implemented as identity authentication using a smart key or a near field communication (NFC) device. Bluetooth pairing verification can be specifically implemented as pairing the Bluetooth of an electronic device such as a smartphone with the Bluetooth of the vehicle. Successful pairing means successful verification.

[0065] Through diverse verification mechanisms such as biometric verification, smart key authority verification, and Bluetooth pairing verification, it is possible to determine whether the target user has the authority to interact with the vehicle outside the vehicle, thereby improving the accuracy of identity authentication and thus the security of interaction.

[0066] S203: When the interaction authority verification of the target user outside the vehicle is passed, voice information of the target user is obtained.

[0067] When the verification result of verifying the interaction authority of the target user indicates that the verification is successful, voice information of the target user is obtained through a microphone array outside the vehicle.

[0068] S204: Perform speech recognition and natural language understanding on the speech information through a speech processing model to determine a speech understanding result corresponding to the speech information.

[0069] In this step, the speech processing model can be used to perform speech recognition and natural language understanding on the speech information, identify key words and sentence structures in the speech information, and thus understand the needs of the target user and obtain speech understanding results.

[0070] Among them, the speech processing model can be not only LLM, but also a lightweight model running locally on the vehicle-mounted device. Running locally can reduce bandwidth occupancy and latency.

[0071] Another option is to use a hybrid cloud solution, running speech processing models locally and in the cloud at the same time, giving priority to the local speech processing model, and using the cloud for recognition when the local recognition accuracy is insufficient or the model is not supported. The hybrid cloud solution can effectively improve the output speed and accuracy of speech understanding results.

[0072] S205. Perform sentiment analysis based on the speech understanding result through the speech processing model to determine the sentiment analysis result corresponding to the speech information.

[0073] In this step, the speech understanding results can be further analyzed through the speech processing model to identify the emotional color of the target user in the speech information, such as calmness, joy, anxiety, sadness, anger, etc., and obtain the sentiment analysis results.

[0074] Among them, the speech processing model can be not only LLM, but also a pre-trained model specifically used for sentiment classification, which may be lighter and less resource-intensive.

[0075] In addition to using speech processing models for sentiment analysis, speech emotion recognition technology can also be used to integrate the characteristics of speech information, such as speech speed and intonation, to perform sentiment analysis, which can improve the accuracy of sentiment analysis. Speech emotion recognition technology combined with speech features can not only understand the target user's language content, but also identify the target user's emotional state based on speech speed, intonation, etc., thereby providing intelligent and humanized interactive responses, and is more adaptable to the situation.

[0076] S206: Determine the conversation content for replying to the voice information based on the voice understanding result and the sentiment analysis result through the voice processing model.

[0077] In this step, the speech processing model can be used to refer to the speech understanding results, i.e., the needs of the target user, and the sentiment analysis results, i.e., the emotional state of the target user, to determine the content of the conversation in reply to the voice message.

[0078] In addition to using a speech processing model to determine the conversation content of a reply voice message, the conversation content of a reply voice message can also be determined with the help of a preset dialogue library. Specifically, different dialogue libraries are set in advance according to different user needs and user emotional states, and in actual application, the conversation content of the reply voice message is determined in the corresponding dialogue library.

[0079] When the speech understanding result indicates that the speech information is directive speech, such as querying the weather, etc., a lighter task-oriented dialogue system can be used to determine the dialogue content of the reply voice message to reduce resource usage.

[0080] S207: Determine a target feedback mode corresponding to the voice information from a plurality of preset feedback modes based on the classification types corresponding to the voice understanding result, the sentiment analysis result, and the conversation content.

[0081] There is no limitation on the classification types of speech understanding results, which may include information query and acquisition, task execution and management, entertainment interaction, emotional support, technical support, etc.

[0082] There is no limitation on the classification type of sentiment analysis results, which may include calm, joy, anxiety, sadness, anger, etc.

[0083] There is no limitation on the classification type of the conversation content. For example, it can be determined based on the classification type of the speech understanding result and the classification type of the sentiment analysis result. It can be a combination of any classification type of the speech understanding result and any classification type of the sentiment analysis result. For example, when the classification type of the speech understanding result is emotional support and the classification type of the sentiment analysis result is sadness, the classification type of the conversation content can be sadness relief corresponding to emotional support.

[0084] In the embodiment of the present invention, multiple preset feedback modes can be pre-set, and the classification type of the speech understanding result, the classification type of the sentiment analysis result, and the classification type of the dialogue content for each preset feedback mode can be set. Through the pre-set information, in actual application, based on the classification types corresponding to the speech understanding result, the sentiment analysis result, and the dialogue content, a target feedback mode corresponding to the speech information can be matched from multiple preset feedback modes.

[0085] Exemplarily, when the classification type of the speech understanding result is emotional support, the classification type of the sentiment analysis result is sadness, and the classification type of the conversation content is emotional support and the corresponding sadness is alleviated, the corresponding target feedback mode may indicate: playing the conversation content of the reply voice message through the external speaker of the vehicle, the conversation content is words to comfort the target user; flashing the external lights of the vehicle in red; and constantly switching the hidden door handle between extended and retracted states.

[0086] Exemplarily, when the classification type of the speech understanding result is emotional support, the classification type of the sentiment analysis result is joy, and the classification type of the conversation content is a joy response corresponding to emotional support, the corresponding target feedback mode may indicate: playing the sound effects corresponding to the emotion of joy through the external speakers of the car, playing the words that positively respond to the target user; and swinging the wipers back and forth.

[0087] S208: Controlling the vehicle external device to perform multimodal feedback based on the target feedback mode, wherein the multimodal feedback at least includes playing the conversation content.

[0088] The technical solution of the embodiment of the present invention improves the security of interaction through identity verification before human-vehicle interaction; by combining voice understanding and emotion recognition for feedback, it can better adapt to the needs and emotional state of target users, enhance the naturalness of human-vehicle interaction, and provide differentiated interaction experiences for different target users; through a variety of feedback methods, such as the actions of vehicle external lights, windows, wipers, vehicle door handles, external speakers, etc., the feedback is more intuitive, and the richness of the interaction is improved. Compared with the traditional single feedback method, it is more humane and interactive, especially in different situations such as emergency situations. It has stronger adaptability.

[0089] The technical solution of the embodiment of the present invention can adopt a distributed control method to enable different vehicle external devices that perform multi-modal interaction to operate independently and cooperate with each other, thereby reducing the complexity of the overall control; a rule engine can be used to process complex multi-conditional control logic, making configuration and maintenance easier. Through distributed control and rule engines, the flexibility, fault tolerance and maintainability of human-vehicle interaction can be increased, and different control modules can operate independently, but the interaction logic can be flexibly adjusted.

[0090] The technical solution of the embodiment of the present invention significantly improves the intelligence level of the vehicle and the user interaction experience by integrating technologies such as identity authentication, sentiment analysis, multimodal interaction, and distributed control, and has good application prospects.

[0091] Embodiment 3

[0092] Figure 3is a schematic diagram of the structure of a vehicle-vehicle interaction device provided by the third embodiment of the present invention. This embodiment can be applied to the situation of realizing vehicle-vehicle interaction outside the vehicle, such as Figure 3 As shown, the specific structure of the device includes:

[0093] An acquisition module 31 is used to acquire voice information of a target user outside the vehicle;

[0094] A speech processing module 32, used to determine the speech understanding result corresponding to the speech information, the sentiment analysis result corresponding to the speech information, and the dialogue content of replying to the speech information through a speech processing model;

[0095] A feedback mode determination module 33, used to determine a target feedback mode corresponding to the speech information based on the speech understanding result, the sentiment analysis result and the conversation content;

[0096] The feedback execution module 34 is used to control the vehicle external device to execute multimodal feedback based on the target feedback mode, and the multimodal feedback at least includes the playback of the dialogue content.

[0097] The vehicle-outside human-vehicle interaction device provided in this embodiment obtains the voice information of the target user outside the vehicle through the acquisition module; determines the voice understanding result corresponding to the voice information, the emotion analysis result corresponding to the voice information and the dialogue content of replying to the voice information through the voice processing module and the voice processing model; determines the target feedback mode corresponding to the voice information based on the voice understanding result, the emotion analysis result and the dialogue content through the feedback mode determination module; controls the vehicle external device to execute multimodal feedback based on the target feedback mode through the feedback execution module, and the multimodal feedback at least includes the playback of the dialogue content. This solution provides a natural, friendly and rich vehicle-outside human-vehicle interaction experience through accurate language understanding, emotion perception and multimodal feedback including dialogue replies. The addition of emotion perception enables the multimodal feedback method to be adjusted according to the emotional state of the target user during interaction, thereby realizing more personalized interactive services.

[0098] Furthermore, the multimodal feedback further includes at least one of the following:

[0099] Changes in the color and flashing pattern of the vehicle's exterior lights; changes in the opening and closing states of the windows; changes in the action of the windshield wipers; changes in the position of the vehicle's door handles; and the playing of sound effects from the vehicle's exterior speakers.

[0100] Wherein, the vehicle external equipment at least includes the vehicle external lights, the vehicle windows, the wipers, the vehicle door handles, and the vehicle external speakers.

[0101] Furthermore, the speech processing module 32 is specifically used for:

[0102] Performing speech recognition and natural language understanding on the speech information through a speech processing model to determine a speech understanding result corresponding to the speech information;

[0103] Performing sentiment analysis based on the speech understanding result through the speech processing model to determine the sentiment analysis result corresponding to the speech information;

[0104] The speech processing model is used to determine the content of the conversation in response to the speech information based on the speech understanding result and the sentiment analysis result.

[0105] Furthermore, the feedback mode determination module 33 is specifically configured to:

[0106] Based on the classification types corresponding to the speech understanding result, the sentiment analysis result and the conversation content, a target feedback mode corresponding to the speech information is determined from a plurality of preset feedback modes.

[0107] Furthermore, the acquisition module 31 is specifically used for:

[0108] When the interaction authority verification of the target user outside the vehicle is passed, voice information of the target user is acquired.

[0109] Furthermore, the device also includes:

[0110] An interactive wake-up module, used for waking up the vehicle-person interaction outside the vehicle in response to an interactive wake-up operation of the target user outside the vehicle before acquiring the voice information of the target user outside the vehicle;

[0111] The authority verification module is used to verify the interaction authority of the target user and feed back the verification result.

[0112] Furthermore, the target user's interaction authority verification method includes at least one of the following:

[0113] Biometric verification, smart key permission verification, Bluetooth pairing verification.

[0114] The vehicle-external human-vehicle interaction device provided in an embodiment of the present invention can execute the vehicle-external human-vehicle interaction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0115] Embodiment 4

[0116] Figure 4 1 is a schematic diagram of a vehicle structure for implementing an embodiment of the present invention. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0117] like Figure 4 As shown, the vehicle 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the vehicle 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0118] A number of components in the vehicle 10 are connected to the I / O interface 15, including: an input unit 16; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the vehicle 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0119] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the vehicle-outside human-vehicle interaction method.

[0120] In some embodiments, the vehicle-outside human-vehicle interaction method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the vehicle 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the vehicle-outside human-vehicle interaction method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the vehicle-outside human-vehicle interaction method in any other appropriate manner (e.g., by means of firmware).

[0121] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0123] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a vehicle having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the vehicle. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0126] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0127] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0128] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A vehicle-vehicle interaction method outside a vehicle, characterized in that: include: Acquire voice information of a target user outside the vehicle; Determine, through a speech processing model, a speech understanding result corresponding to the speech information, a sentiment analysis result corresponding to the speech information, and a conversation content in response to the speech information; Determining a target feedback mode corresponding to the speech information based on the speech understanding result, the sentiment analysis result and the conversation content; Based on the target feedback mode, the vehicle external device is controlled to perform multimodal feedback, and the multimodal feedback at least includes playing the conversation content.

2. The method according to claim 1, characterized in that The multimodal feedback further includes at least one of the following: Changes in the color and flashing pattern of the vehicle's exterior lights; changes in the opening and closing states of the windows; changes in the action of the wipers; changes in the position of the vehicle's door handles; Playback of sound effects from external speakers of the vehicle; Wherein, the vehicle external equipment at least includes the vehicle external lights, the vehicle windows, the wipers, the vehicle door handles, and the vehicle external speakers.

3. The method according to claim 1, characterized in that Determining, by means of a speech processing model, a speech understanding result corresponding to the speech information, a sentiment analysis result corresponding to the speech information, and a conversation content in response to the speech information, including: Performing speech recognition and natural language understanding on the speech information through a speech processing model to determine a speech understanding result corresponding to the speech information; Performing sentiment analysis based on the speech understanding result through the speech processing model to determine the sentiment analysis result corresponding to the speech information; The speech processing model is used to determine the content of the conversation in response to the speech information based on the speech understanding result and the sentiment analysis result.

4. The method according to claim 1, characterized in that: Determining a target feedback mode corresponding to the voice information based on the voice understanding result, the sentiment analysis result, and the conversation content includes: Based on the classification types corresponding to the speech understanding result, the sentiment analysis result and the conversation content, a target feedback mode corresponding to the speech information is determined from a plurality of preset feedback modes.

5. The method according to claim 1, characterized in that Acquire the voice information of the target user outside the vehicle, including: When the interaction authority verification of the target user outside the vehicle is passed, voice information of the target user is acquired.

6. The method according to claim 1, characterized in that Before acquiring the voice information of the target user outside the vehicle, the method further includes: In response to an interactive awakening operation of a target user outside the vehicle, awakening a human-vehicle interaction outside the vehicle; The interaction authority of the target user is verified and the verification result is fed back.

7. The method according to claim 5 or 6, characterized in that: The target user's interactive authority verification method includes at least one of the following: Biometric verification, smart key permission verification, Bluetooth pairing verification.

8. A vehicle-to-vehicle interaction device outside a vehicle, characterized in that: include: An acquisition module, used for acquiring voice information of a target user outside the vehicle; A speech processing module, used to determine the speech understanding result corresponding to the speech information, the sentiment analysis result corresponding to the speech information, and the dialogue content of replying to the speech information through a speech processing model; A feedback mode determination module, used to determine a target feedback mode corresponding to the speech information based on the speech understanding result, the sentiment analysis result and the conversation content; A feedback execution module is used to control the vehicle external device to execute multimodal feedback based on the target feedback mode, and the multimodal feedback at least includes the playback of the dialogue content.

9. A vehicle, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images