Vehicle front environment recognition and voice interaction method and system

Abstract

This invention provides a vehicle forward environment recognition and voice interaction method, belonging to the field of automotive technology. Responding to user intent commands, it acquires vehicle position and direction data; constructs a target list along the vehicle's main line of sight based on the vehicle position and direction data; filters targets from the target list based on image information along the vehicle's main line of sight; and provides feedback on the user's intent commands based on the filtered targets. This invention's vehicle forward environment recognition and voice interaction method integrates GPS positioning, vehicle camera visual information, direction perception, and AI large-scale model multimodal analysis capabilities into an intelligent recognition system. It can respond to user voice commands, recognize and introduce buildings or scenery in front of the vehicle in real time, and provide accurate, rich, and natural language feedback.

Description

Technical Field

[0001] This invention relates to the field of automotive technology, and in particular to a method and system for vehicle front environment recognition and voice interaction. Background Technology

[0002] With the rapid development of intelligent connected vehicles and in-vehicle human-machine interaction systems, users' demand for environmental information during driving is increasing. Traditional in-vehicle navigation systems mainly rely on map data to provide route guidance, but lack the ability to analyze the "seen" scene in real time. In recent years, some high-end models have begun to integrate voice assistants and cameras to try to achieve "image-based communication" functionality, but problems such as low recognition accuracy, weak semantic understanding, and unnatural feedback still exist.

[0003] In existing technologies, the system locates the vehicle's position using GPS and combines this with a high-precision map database to identify preset POI (Point of Interest) information near the vehicle. When the vehicle approaches a landmark, the system automatically plays pre-recorded voice or text-to-speech to announce the information. However, this system relies too heavily on preset data and judges the location solely based on location, without considering the actual field of vision, which may lead to false alarms such as "seeing A but announcing B". Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes a method and system for vehicle front environment recognition and voice interaction.

[0005] In a first aspect, embodiments of the present invention provide a method for vehicle front environment recognition and voice interaction, comprising:

[0006] In response to user intent commands, obtain vehicle position and orientation data;

[0007] Based on the vehicle position and the direction data, construct a target list in the vehicle's main line-of-sight direction;

[0008] Targets are filtered from the target list based on image information from the vehicle's main line of sight.

[0009] Feedback on user intent commands based on the selected target.

[0010] In some embodiments, the response to a user intent instruction includes:

[0011] Capture user voice commands;

[0012] Convert the preprocessed voice commands into text information;

[0013] The user's intent is analyzed based on the text information.

[0014] In some embodiments, obtaining the target list in the vehicle's main line-of-sight direction based on the vehicle's position and the direction data includes:

[0015] Based on the vehicle's location, call the map service API to obtain all Points of Interest (POIs) within a set range of the vehicle's location, and obtain an initial target list;

[0016] Based on vehicle orientation data, a list of targets in the vehicle's main line of sight is filtered from the initial target list.

[0017] In some embodiments, filtering targets from the target list based on image information from the vehicle's line of sight includes:

[0018] Extract the metadata of the target list, spatially align and visually match the image in the vehicle's main line of sight with the metadata, and obtain the matched target.

[0019] In some embodiments, the user intent instruction based on the filtered target feedback includes:

[0020] The image information of the main line of sight, the metadata of the target to be filtered, the vehicle position, and the vehicle direction data are input into the visual-language multimodal large model, and the target to be filtered is confirmed through the input visual-language multimodal large model;

[0021] The visual-language multimodal big model encodes image information, vehicle position, vehicle orientation data, and metadata of the selected target in a unified manner and generates natural language text.

[0022] In some embodiments, the user intent instruction based on the filtered target feedback includes:

[0023] It calls the TTS engine and converts natural language text into a speech audio stream based on preset or user-selected speech styles;

[0024] The audio stream is transmitted to the car audio system.

[0025] In some embodiments, the user intent instruction based on the filtered target feedback includes:

[0026] Label the target with a name in the header display.

[0027] Secondly, embodiments of the present invention provide a vehicle forward environment recognition and voice interaction system, configured to implement the above method, including:

[0028] The data acquisition unit responds to user intent commands and acquires vehicle position and orientation data;

[0029] The construction unit constructs a target list in the vehicle's main line-of-sight direction based on the vehicle's position and direction data.

[0030] The filtering unit filters targets from the target list based on image information from the vehicle's main line-of-sight direction.

[0031] The feedback unit provides feedback on the user's intent commands based on the selected target.

[0032] Thirdly, embodiments of the present invention also provide an electronic device, comprising:

[0033] One or more processors;

[0034] Memory, used to store one or more programs;

[0035] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods.

[0036] Fourthly, embodiments of the present invention also provide a computer-readable medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described.

[0037] The present invention provides a vehicle forward environment recognition and voice interaction method that simultaneously acquires vehicle position and direction data based on the user's voice input and intent command; based on the vehicle position and direction data, it identifies a list of possible targets in front of the vehicle; and it filters targets based on vision and provides feedback on the user's intent command. This method integrates GPS positioning, vehicle camera visual information, direction perception, and AI large-scale model multimodal analysis capabilities into an intelligent recognition system. This system can respond to user voice commands, identify and describe buildings or scenery in front of the vehicle in real time, and provide accurate, rich, and natural language feedback. Attached Figure Description

[0038] Figure 1 This is a schematic diagram illustrating the steps of an embodiment of the vehicle front environment recognition and voice interaction method of the present invention;

[0039] Figure 2 This is a schematic diagram illustrating the steps of an embodiment of the present invention that outputs user intent commands based on user voice.

[0040] Figure 3 This is a schematic diagram of the structure of an embodiment of the vehicle front environment recognition and voice interaction system of the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0043] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0044] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0045] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0046] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0047] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0048] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a method for vehicle front environment recognition and voice interaction. Figure 1 A flowchart illustrating the steps of a vehicle front environment recognition and voice interaction method provided in an embodiment of the present invention.

[0049] like Figure 1 As shown, the vehicle's forward environment recognition and voice interaction method includes the following steps:

[0050] Step S10: In response to the user's intent command, obtain vehicle position and orientation data.

[0051] Please see Figure 2 This embodiment requires clearly capturing the user's voice commands in a complex in-vehicle acoustic environment.

[0052] The specific implementation method is as follows:

[0053] Four high-sensitivity microphones are strategically placed inside the vehicle to form a microphone array (typically located near the steering wheel, rearview mirror, or roof). Each microphone simultaneously collects audio signals and transmits them synchronously to the voice processing unit via an audio interface.

[0054] An adaptive filter is used to eliminate echoes from the microphone signal, using the sound played from the vehicle's speakers as a reference signal. The delay and weight of each microphone are calculated based on sound source localization (assuming a fixed user position or location via a camera), and the beam is directed towards the user while suppressing noise from other directions. The enhanced signal is then framed, windowed, and acoustic features (such as MFCC) are extracted for use in ASR (Automatic Speech Recognition).

[0055] Convert the preprocessed audio signal into text information.

[0056] The specific implementation method is as follows:

[0057] Depending on network conditions and voice length, a local engine (such as iFlytek's offline engine) or a cloud engine (such as Alibaba Cloud ASR) is selected to convert the audio signal into an acoustic feature vector sequence. Using acoustic models, language models, and dictionaries, the feature vector sequence is then converted into the most probable text sequence, outputting text information and confidence levels.

[0058] Extract key information from text messages to determine user intent.

[0059] The specific implementation method is as follows:

[0060] Use a classification model (such as a BERT-based model) to classify the text and determine the intent category (e.g., "identify building ahead"). Use a sequence labeling model (such as BiLSTM-CRF) or an end-to-end model to extract key information from the text, such as "building" as the object and "ahead" as the location, and combine the intent and slots into a structured instruction, for example: {"intent": "identify building", "target": "building", "direction": "ahead"}.

[0061] Furthermore, it supports context-sensitive continuous dialogue.

[0062] Implementation methods include:

[0063] Maintain a dialogue state, recording the topic, mentioned entities, and attributes of the current conversation. When the user uses pronouns (such as "this" or "it"), determine the entity they refer to based on the dialogue history. Combine the intent and slot of the current conversation with the historical dialogue state to form a complete user command.

[0064] For example, if a user first asks, "Can you help me identify what the building in front of me is?", the system identifies it as building A and answers. If the user then asks, "How tall is it?", the system uses the denotation function to determine that "it" refers to building A, and then queries the height of building A.

[0065] In addition, it can be understood that, as the perception layer of the system, upon receiving the user's intention command, it immediately and synchronously acquires the geographic space where the vehicle is located and the visual scene data in front, providing raw input for subsequent fusion analysis.

[0066] The specific implementation method is as follows:

[0067] Synchronously obtain vehicle location: Read the vehicle's current latitude and longitude coordinates from the onboard GPS module.

[0068] Synchronous acquisition of directional data: Read the vehicle's heading angle from the IMU (Inertial Measurement Unit) to accurately determine the vehicle's orientation.

[0069] In addition, the vehicle's forward-facing high-definition camera is used to capture real-time images of the front with a resolution of at least 1080p and a wide field of view (FOV≥90°).

[0070] It is understandable that side-view or panoramic camera images can be used as needed to help determine whether the target is partially obscured.

[0071] For example, upon receiving a user intent command, output a set of time-aligned multi-data packets, including: {GPS coordinates, heading angle, and forward image}.

[0072] Understandably, synchronization is crucial. Ensuring that the geographical location, vehicle orientation, and image information are highly consistent in time is the foundation for accurate spatial matching later.

[0073] Step S20: Based on the vehicle position and the direction data, construct a target list in the vehicle's main line of sight direction.

[0074] Understandably, as a spatial information filter, this step associates the abstract vehicle location and orientation data with a real-world geographic information database (POI database) to initially identify a list of possible targets in the vehicle's main line of sight, greatly narrowing the search scope for subsequent AI models.

[0075] The specific implementation method is as follows:

[0076] POI Search: Using the vehicle's GPS coordinates as the center, the API (Application Programming Interface) of map services such as Gaode / Google Maps is called to search for all points of interest (POIs) within a radius (e.g., 500 meters) to obtain an initial list of targets containing information such as name, category, latitude and longitude, and description.

[0077] Directional filtering: Using the vehicle's heading angle, a fan-shaped visible area is calculated, 30 degrees to the left and right of the vehicle's heading. Through geometric calculations, a target list is formed from the initial target list, selecting targets falling within this fan-shaped geographic area in the vehicle's primary line of sight. For example, 15 POIs located "directly in front" are selected from 200 POIs to form the target list.

[0078] Step S30: Filter targets from the target list based on image information from the vehicle's main line of sight.

[0079] Preliminary visual association: The metadata of each POI in the target list, such as its name and distinctive features, is quickly matched with image information in the opposite direction of the vehicle's main line of sight. For example, if a prominent tower-like structure is identified in the image, and "TV tower" is a POI in the target list, its priority is increased.

[0080] It is understandable that the output of this step, as prior knowledge or context, is input into the next step along with the original preceding image, guiding the multimodal large model to perform focused and efficient reasoning.

[0081] Step S40: Feedback user intent instructions based on the filtered target.

[0082] As the core of the system's intelligent decision-making, this step receives all information from the perception layer, performs deep cross-modal understanding and reasoning, and ultimately generates natural language answers to user questions.

[0083] The specific implementation method is as follows:

[0084] Multimodal input construction: The vehicle position and orientation data output in step S10, as well as the image information of the vehicle's main line of sight, and the metadata of the filtered target output in step S30 are combined.

[0085] Model inference: The integrated prompts are input into a pre-trained visual-language multimodal large model (such as Qwen-VL, GPT-4V), and the model performs the following sub-tasks:

[0086] The system deeply understands image content, recognizing visual elements such as architectural style, color, shape, and signage. It then correlates, compares, and verifies the visual analysis results with the metadata of the selected targets to confirm their identities. Based on the confirmed target identity, and combining general knowledge embedded in the model with external knowledge retrieved in real time, it organizes a complete and accurate description in natural language to directly respond to the user's intent commands.

[0087] The visual-language multimodal large model outputs natural language text, which can be further converted into speech information and played through the car audio system to provide feedback on the user's intention commands.

[0088] The specific implementation method is as follows:

[0089] Speech Synthesis: Receives natural language text output from a large visual-language multimodal model, calls a high-quality TTS engine, and synthesizes the natural language text into a fluent and natural audio stream based on preset or user-selected speech styles (timbre, speech rate, intonation parameters, etc.), and transmits the synthesized audio stream to the car audio system.

[0090] It is understandable that the intelligent results of the system should be delivered to the user in a way that best suits the in-vehicle scenario and does not affect driving safety (auditory channel), forming a perfect closed loop from "user asks" to "system answers".

[0091] In the preferred embodiment, when the user receives feedback and asks follow-up questions, the new question will be reasoned again using a visual-language multimodal model, taking into account historical context (such as buildings already identified), thus enabling multi-turn dialogue. In this embodiment, a state loop is added to the core process to achieve continuous contextual interaction.

[0092] In one embodiment, when the vehicle has an AR-HUD display function, the visual-language multimodal large model, after confirming the target, projects the confirmed target from the image information onto the AR-HUD and simultaneously labels the name at the selected target. In this embodiment, in addition to voice output, a parallel visual output channel is added to provide a more intuitive augmented reality experience.

[0093] Understandably, by using GPS, direction, and visual image triple verification, misjudgments from a single data source are avoided. Especially in densely populated urban areas, it can accurately distinguish adjacent buildings, significantly improving recognition accuracy and achieving a leap from "image classification" to "scene understanding." It can answer complex questions and provide in-depth information such as history, culture, and design, enhancing semantic understanding capabilities. It supports natural language questioning and answering, improving user experience and is suitable for various scenarios such as tourism, commuting, and education, with natural and smooth interaction. It can be adapted to different map services, AI models, and in-vehicle hardware platforms, supporting continuous iteration and upgrades.

[0094] The vehicle forward environment recognition and voice interaction method provided by this invention outputs user intent commands based on user voice; simultaneously acquires vehicle position and direction data based on the intent commands; locks a list of possible targets ahead of the user based on the vehicle position and direction data; generates a final text response based on the target list; and feeds the final text response back to the user. This vehicle forward environment recognition and voice interaction method, which integrates GPS positioning, vehicle camera visual information, direction perception, and AI large-scale model multimodal analysis capabilities into an intelligent recognition system, can respond to user voice commands, identify and introduce buildings or scenery ahead of the vehicle in real time, and provide accurate, rich, and natural language feedback.

[0095] Based on the same inventive concept, the present invention also provides a vehicle front environment recognition and voice interaction system. Figure 3 This is a schematic diagram of a vehicle front environment recognition and voice interaction system provided in an embodiment of the present invention. It is applied to the vehicle front environment recognition and voice interaction method provided in the above embodiment and specifically includes: a data acquisition unit, a data construction unit, a data filtering unit, and a feedback unit.

[0096] The data acquisition unit responds to user intent commands and acquires vehicle position and orientation data.

[0097] It is understandable that, as the perception layer of the system, upon receiving the user's intent command, it immediately and synchronously acquires the geographic spatial status of the vehicle and the visual scene data in front, providing raw input for subsequent fusion analysis.

[0098] The specific implementation method is as follows:

[0099] Synchronously obtain vehicle location: Read the vehicle's current latitude and longitude coordinates from the onboard GPS module.

[0100] Synchronous acquisition of directional data: Read the vehicle's heading angle from the IMU (Inertial Measurement Unit) to accurately determine the vehicle's orientation.

[0101] In addition, the vehicle's front-view high-definition camera is used to capture real-time images of the front with a resolution of at least 1080p and a wide field of view (FOV≥90°), specifically obtaining the vehicle's main line of sight image information.

[0102] It is understandable that side-view or panoramic camera images can be used as needed to help determine whether the target is partially obscured.

[0103] Understandably, synchronization is crucial. Ensuring that the geographical location, vehicle orientation, and image information are highly consistent in time is the foundation for accurate spatial matching later.

[0104] In addition, for user intent commands, the user's voice information can be collected to generate corresponding intent commands.

[0105] As a result, the acquisition unit can clearly capture the user's voice commands even in the complex acoustic environment inside the vehicle.

[0106] The specific implementation method is as follows:

[0107] Four high-sensitivity microphones are strategically placed inside the vehicle to form a microphone array (typically located near the steering wheel, rearview mirror, or roof). Each microphone simultaneously collects audio signals and transmits them synchronously to the voice processing unit via an audio interface.

[0108] The speech processing unit converts the preprocessed audio signal into text information, and then parses key information from the text information to determine the user's intention command.

[0109] The construction unit constructs a target list in the vehicle's main line-of-sight direction based on the vehicle's position and direction data.

[0110] As a spatial information filter, this module associates abstract vehicle location and orientation data with a real-world geographic information database (POI database) to construct all possible points of interest (POIs) in the vehicle's main line of sight. Each POI contains information such as name, category, latitude and longitude, and description to form a target list, greatly narrowing the target selection range.

[0111] The filtering unit filters targets from the target list based on image information from the vehicle's main line of sight.

[0112] Visual association: Quickly match the textual information such as the name and distinctive features of all POIs in the target list with image information in the direction of the vehicle's main line of sight. For example, if a prominent tower-like structure is identified in the image, and "TV tower" is among the candidate POIs, its priority is increased.

[0113] The feedback unit provides feedback on the user's intent commands based on the selected target.

[0114] As the core of the system's intelligent decision-making, this module receives all information from the perception layer, performs deep cross-modal understanding and reasoning, and ultimately generates natural language answers to user questions.

[0115] The feedback unit includes a visual-language multimodal large model, a voice playback module, and a video playback module.

[0116] The visual-language multimodal large model deeply understands image content and identifies visual elements such as architectural style, color, shape, and text signage. The visual analysis results are correlated, compared, and verified with the metadata of the selected targets to confirm their identities. Based on the confirmed target identities, and combined with general knowledge embedded in the model or external knowledge retrieved in real time, natural language text is output.

[0117] The voice playback module can serve as the system's interactive interface, converting the natural language text output by the visual-language multimodal model into voice information and playing it through the car's audio system to provide feedback on the user's intent commands.

[0118] The video playback module can also serve as the system's interactive exit point. After confirming the target, the visual-language multimodal large model projects the confirmed target from the image information through AR-HUD and labels the name at the target.

[0119] Understandably, the triple verification of GPS, direction, and visual images avoids misjudgments from a single data source, especially in densely populated urban areas where it can accurately distinguish adjacent buildings, significantly improving recognition accuracy. The introduction of a large AI model enables a leap from "image classification" to "scene understanding," capable of answering complex questions and providing in-depth information such as history, culture, and design, enhancing semantic understanding capabilities. It supports natural language questioning and answering, improving user experience and is suitable for various scenarios such as tourism, commuting, and education, with natural and smooth interaction. It is adaptable to different map services, AI models, and in-vehicle hardware platforms, supporting continuous iteration and upgrades with strong scalability. It prioritizes using local lightweight models for simple requests, only calling large cloud models for complex tasks, balancing response speed and computational overhead, thus optimizing resources.

[0120] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 4As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement any of the vehicle forward environment recognition and voice interaction methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0121] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0122] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0123] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0124] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the vehicle forward environment recognition and voice interaction methods described in the above embodiments. The computer-readable storage medium can be volatile or non-volatile.

[0125] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described vehicle front environment recognition and voice interaction method.

[0126] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0127] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0128] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0129] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0130] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0131] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0135] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A method for vehicle front environment recognition and voice interaction, characterized in that, It includes: In response to user intent commands, obtain vehicle position and orientation data; Based on the vehicle position and the direction data, construct a target list in the vehicle's main line-of-sight direction; Targets are filtered from the target list based on image information from the vehicle's main line of sight. Feedback on user intent commands based on the selected target.

2. The vehicle front environment recognition and voice interaction method according to claim 1, characterized in that, The response to the user intent instruction includes: Capture user voice commands; Convert the preprocessed voice commands into text information; The user's intent is analyzed based on the text information.

3. The vehicle front environment recognition and voice interaction method according to claim 1, characterized in that, The step of obtaining a target list in the vehicle's main line-of-sight direction based on the vehicle's position and direction data includes: Based on the vehicle's location, call the map service API to obtain all Points of Interest (POIs) within a set range of the vehicle's location, and obtain an initial target list; Based on vehicle orientation data, a list of targets in the vehicle's main line of sight is filtered from the initial target list.

4. The vehicle front environment recognition and voice interaction method according to claim 1, characterized in that, The process of filtering targets from the target list based on image information from the vehicle's line of sight includes: Extract the metadata of the target list, spatially align and visually match the image in the vehicle's main line of sight with the metadata, and obtain the matched target.

5. The vehicle front environment recognition and voice interaction method according to claim 1, characterized in that, The user intent instruction based on the filtered target feedback includes: The image information of the main line of sight, the metadata of the target to be filtered, the vehicle position, and the vehicle direction data are input into the visual-language multimodal large model, and the target to be filtered is confirmed through the input visual-language multimodal large model; The visual-language multimodal big model encodes image information, vehicle position, vehicle orientation data, and metadata of the selected target in a unified manner and generates natural language text.

6. The vehicle front environment recognition and voice interaction method according to claim 5, characterized in that, The user intent instruction based on the filtered target feedback includes: It calls the TTS engine and converts natural language text into a speech audio stream based on preset or user-selected speech styles; The audio stream is transmitted to the car audio system.

7. The vehicle front environment recognition and voice interaction method according to claim 1, characterized in that, The user intent instruction based on the filtered target feedback includes: Label the target with a name in the header display.

8. A vehicle front environment recognition and voice interaction system, characterized in that, Configured for implementing the method of any one of claims 1-7, comprising: The data acquisition unit responds to user intent commands and acquires vehicle position and orientation data; The construction unit constructs a target list in the vehicle's main line-of-sight direction based on the vehicle's position and direction data. The filtering unit filters targets from the target list based on image information from the vehicle's main line-of-sight direction. The feedback unit provides feedback on the user's intent commands based on the selected target.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images