A natural language-based in-vehicle broadcast interaction method and system

Abstract

This invention belongs to the field of vehicle-mounted human-machine interaction technology, specifically relating to a vehicle-mounted broadcast interaction method and system based on natural language. It involves acquiring broadcast audio streams, understanding the broadcast content to obtain structured content data, and associating and storing the structured content data with corresponding timestamps to construct a broadcast content time index library. When a user's verbal command is detected, speech recognition and natural language understanding are performed to parse the user's intent and time indication information, thereby determining a backtracking time window and quickly retrieving the corresponding structured content data using the backtracking time window as an index. Finally, based on the user's intent, corresponding target operations are executed and / or response content is generated, and multimodal feedback is performed. This enables interactive operations such as replay, query, or information retrieval, transforming the broadcast from a one-way receiving device into a conversational intelligent terminal. The elimination of manual operation significantly reduces the risk of driver distraction and improves driving safety and information acquisition efficiency.

Description

Technical Field

[0001] This invention belongs to the field of vehicle-mounted human-machine interaction technology, specifically relating to a vehicle-mounted broadcast interaction method and system based on natural language. Background Technology

[0002] Current in-vehicle radio systems (FM / AM / DAB) only provide one-way audio streaming, so users can only listen passively and cannot interact with the broadcast content in any way. They can only switch broadcast channels based on voice assistants, and the broadcast system is still in an information silo state.

[0003] In everyday driving scenarios, traditional in-vehicle broadcasting systems cannot meet users' needs for real-time information retrieval and content interaction, specifically as follows: Music listening scenario: When listening to songs, users often fail to obtain information about the song being played because the host's announcement of the song title is unclear or they miss the information. When users want to ask "What is the name of this song?" or "Add this song to their favorites", the car's radio system cannot respond and can only use the phone's music recognition function. This requires going through the process of "taking out the phone - opening the app - recognizing - manually adding to favorites", which distracts the driver's attention and poses a safety hazard.

[0004] News broadcast scenario: After broadcasting important policy information (such as "new energy subsidies will be reduced by 30%), if users do not remember the key data (such as the reduction rate) immediately, and try to ask "how much did the subsidy just say decrease?" a few minutes later, the broadcast content has disappeared, and content recall is not supported, so users cannot obtain the information that was previously broadcast.

[0005] Radio station switching scenario: After switching between different radio stations, if a user wants to retrieve information about the song that was just played on the previous channel (such as "What was the name of that song?"), traditional broadcast systems lose all context when switching stations and cannot provide any historical content query capabilities.

[0006] Interview program listening scenario: In an interview program, when the host mentions a technical term (such as "L3 standard"), and the user later wants to ask "What is the L3 standard that the host just mentioned?", the system cannot identify the program role referred to by "host", nor can it understand the time range corresponding to "just", lacking the ability to understand and index the semantics of the broadcast content. Summary of the Invention

[0007] The purpose of this invention is to disclose a natural language-based in-vehicle broadcasting interaction method and system, which solves the technical problem that existing in-vehicle broadcasting systems cannot meet users' interactive needs for tracing and querying broadcast content.

[0008] To achieve the above objectives, this invention discloses a natural language-based in-vehicle broadcast interaction method, comprising: The broadcast audio stream is collected, the broadcast content is understood to obtain structured content data, the structured content data is associated with the corresponding timestamps and stored, and a broadcast content time index library is built. The system acquires user language commands, performs speech recognition and natural language understanding, parses user intent and time indication information, and then determines the backtracking time window based on the time indication information. Using the aforementioned retrospective time window as an index, the corresponding structured content data is retrieved from the broadcast content time index library; Based on the user's intent, the structured content data is processed to perform the corresponding target operation and / or generate response content and perform feedback.

[0009] This basic solution acquires broadcast audio streams, performs content understanding to obtain structured content data, associates and stores the structured content data with corresponding timestamps, and constructs a broadcast content time index library, transforming unstructured audio streams into time-stamped structured data. When a user's verbal command is detected, speech recognition and natural language understanding are performed to parse the user's intent and time indication information, thereby determining the backtracking time window and quickly retrieving the corresponding structured content data using the backtracking time window as an index. Finally, based on the user's intent, the structured content data is processed to execute corresponding target operations and / or generate response content and provide feedback. This enables interactive operations such as replay, query, or information retrieval (i.e., supporting rich interactive functions such as music collection, news playback, station switching recall, and role identification), transforming broadcasting from a one-way receiving device into a conversational intelligent terminal. The elimination of manual operation significantly reduces the risk of driver distraction, improves driving safety and information acquisition efficiency, and provides users with a good in-vehicle radio listening experience.

[0010] As an optional implementation, broadcast audio streams are acquired, broadcast content is understood to obtain structured content data, and the structured content data is associated with and stored with corresponding timestamps to construct a broadcast content time index library, including: The broadcast audio stream is collected in parallel between the digital audio output terminal of the car radio tuner and the power amplifier input terminal through a high-impedance differential probe; Timestamp synchronization involves adding a synchronization frame containing the current time information to the broadcast audio stream to obtain the target audio stream. Multimodal content recognition is performed on the target audio stream, and broadcast frequency switching events are detected in real time to preserve context, thereby integrating the data to obtain structured content data. Obtain the synchronization frame corresponding to the structured content data, then determine the corresponding timestamp and associate it with the structured content data for storage, generating a broadcast content time index library.

[0011] This solution acquires broadcast audio streams via a high-impedance differential probe connected in parallel between the digital audio output of the car radio tuner and the input of the power amplifier. This allows for zero-latency, zero-loss acquisition of audio perfectly synchronized with the user's audio without interrupting the original playback path. A synchronization frame containing current time information is added to the broadcast audio stream to obtain the target audio stream. Hardware timestamp synchronization ensures that each audio segment has a unique and precise time identifier, allowing users to accurately query and rewind audio as needed.

[0012] As an optional implementation, multimodal content recognition is performed on the target audio stream, including: Based on streaming speech recognition, the target audio stream is converted into ASR transcribed text, and the start and end timestamps of each sentence are recorded. Based on music fingerprint recognition, the metadata of songs in the target audio stream is identified, and the timestamp of the identification time is recorded; The target audio stream is structured using speaker log technology, the speaker audio is segmented, and it is categorized based on role tags.

[0013] This solution applies multimodal content recognition technologies (streaming speech recognition, music fingerprinting, speaker logs, etc.) to the target audio stream, transforming unstructured audio signals into structured, searchable, and semantically rich structured content data in real time. This enables multi-dimensional content parsing and understanding, allowing for simultaneous understanding of audio content from different dimensions. This supports efficient retrieval, accurate content location, and precise response to user language commands, thus meeting user interaction needs.

[0014] As an optional implementation, real-time detection of broadcast frequency switching events to preserve context includes: When a broadcast frequency switching event is detected, the target audio segment is obtained from the broadcast audio stream before the switching, based on the system time at the time of the switching. Feature extraction is performed on the target audio segment to generate an audio feature vector; Obtain the audio information and audio feature vector of the target audio segment, and save it as a history record to preserve the context; The broadcast content time index library includes one or more of the following: music fingerprint recognition results, ASR transcribed text, speaker log results, and audio feature vectors.

[0015] This solution determines the switching time of the broadcast frequency switching event based on the system time, then extracts the broadcast audio stream before the switch, obtains the corresponding audio information and audio feature vector, and saves it as a historical content record to preserve the context. By preserving the audio features before the switch, it is possible to trace back across frequencies.

[0016] As an optional implementation, the method involves acquiring user language commands, performing speech recognition and natural language understanding, parsing user intent and time indication information, and then determining a backtracking time window based on the time indication information, including: Collect user voice and recognize it to obtain user language commands; Natural language understanding is used to interpret user commands and extract user intent and time indication information. The time indication information is semantically parsed to identify the specific time direction and range boundary, and the corresponding backtracking time window is dynamically calculated.

[0017] This solution collects users' natural language commands, performs speech recognition and natural language understanding, and parses out user intent and time indication information. This allows for interactive operation without manual intervention. Voice interaction control effectively avoids the risk of blind driving caused by distracted screen operation, greatly improving driving safety.

[0018] As an optional implementation, using the backtracking time window as an index, the corresponding structured content data is retrieved from the broadcast content time index library, including: Using the start and end time nodes of the backtracking time window as the retrieval boundary, the broadcast content time index library is quickly retrieved to obtain all structured content data within this time period. Traverse all structured content data, perform semantic similarity calculation on the ASR transcribed text based on keyword matching or using a vector model, and select and output the structured content data with the highest content matching degree with the user's language instruction.

[0019] This solution uses the start and end times of the retrospective time window as the retrieval boundary to quickly search the broadcast content time index library, achieving millisecond-level timestamp positioning; and by combining keyword matching and semantic similarity calculation, it retrieves the structured content data with the highest content matching degree, which can accurately locate the content referred to by the user.

[0020] As an optional implementation, the execution feedback includes: Obtain the current driving status and identify the vehicle's driving load; Pre-map one-to-one corresponding driving load levels and feedback modes, and then match the corresponding feedback modes according to the driving load; Feedback priorities are determined based on the urgency of the target action / response content, and then feedback is executed based on the feedback pattern and user historical preferences.

[0021] This solution identifies the vehicle's driving load in real time by acquiring the current driving status. In high-load scenarios, it automatically suppresses complex visual feedback and switches to concise voice commands to avoid distracting the driver. In low-load scenarios, it provides rich and intuitive multimodal feedback to improve the interactive experience. By combining the urgency of information with feedback priority, it ensures that critical information (mainly related to safe driving) is delivered to the driver promptly, while non-urgent entertainment information is weakened or delayed, reducing cognitive overload and optimizing interaction efficiency. Multimodal feedback is dynamically executed based on the user's historical preferences (such as preference for voice or dashboard prompts), making the interaction more natural and achieving a personalized experience.

[0022] As an optional implementation, the execution of the target operation includes: If the user's intention is to collect songs, extract the song metadata from the structured content data and write it into the user's personal playlist database; If the user's intent is to query content, the answer text is generated by calling the local knowledge base and combining the structured content data, with the keywords in the user's language command as the core. If the user's intent is to play audio, then the corresponding audio segment is extracted from the structured content data and pushed to the player.

[0023] This solution uses structured content data obtained from broadcast audio streams to seamlessly translate natural language commands into specific backend business actions, thereby improving interaction efficiency and user experience.

[0024] As an optional implementation, the feedback mode includes one or more of the following: voice broadcast, tactile vibration, graphic card display, highlighted original text display, playlist, and progress bar anchor point display.

[0025] This solution provides multimodal adaptive feedback, with voice broadcasting, tactile vibration, graphic card display, highlighted original text display, progress bar anchor point display, and other multi-directional collaboration, effectively improving the efficiency and accuracy of information reception and meeting the needs of personalization and scene adaptation.

[0026] The present invention also provides a vehicle-mounted radio interactive system based on natural language, for implementing a vehicle-mounted radio interactive method based on natural language as described above, including a time synchronization module, an audio acquisition module, a content recognition module, an index storage module connected in sequence, and a voice interaction module, a backtracking calculation module, and a retrieval execution module connected in sequence, and further including a feedback module connected to the retrieval execution module, and the index storage module is also connected to the retrieval execution module; The time synchronization module is used to perform timestamp synchronization and output a synchronization frame including the current time information; The audio acquisition module is used to acquire broadcast audio streams and add synchronization frames to output the target audio stream; The content recognition module is used to perform broadcast content understanding on the target audio stream to obtain structured content data; The index storage module is used to associate and store the structured content data with the corresponding timestamps to build a broadcast content time index library; The voice interaction module is used to acquire user language commands, perform speech recognition and natural language understanding, and parse user intent and time indication information; The backtracking calculation module is used to determine the backtracking time window based on the time indication information; The retrieval execution module is used to retrieve the corresponding structured content data from the broadcast content time index library using the backtracking time window as an index, and to process the structured content data based on the user intent in order to perform the corresponding target operation and / or generate response content. The feedback module is used to perform feedback.

[0027] This invention constructs a vehicle-mounted radio interaction system based on natural language, providing a foundation for radio interaction. This allows users to respond instantly, accurately trace, and manage radio content through everyday conversational commands (such as "add this song to your favorites", "how much the subsidy reduction was mentioned earlier", "the standards mentioned by the host"). The entire process requires no manual operation, no application switching, and no reliance on the network, making radio an "interactive" part of the smart cockpit. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating a natural language-based in-vehicle broadcast interaction method provided in an embodiment of the present invention. Figure 2 This is a system framework diagram of a vehicle-mounted broadcast interaction system based on natural language, provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] In this invention, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing the invention and its embodiments, and are not intended to limit the indicated devices, elements, or components to having a specific orientation, or to be constructed and operated in a specific orientation.

[0032] Furthermore, some of the aforementioned terms, besides indicating direction or positional relationships, may also have other meanings. For example, the term "above" may, in certain circumstances, indicate a dependency or connection. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0033] Furthermore, the terms "installation," "setup," "equipped with," "connection," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0034] Furthermore, the terms "first," "second," etc., are primarily used to distinguish different devices, components, or parts (which may be the same or different in specific type and construction), and are not intended to indicate or imply the relative importance or quantity of the indicated devices, components, or parts. Unless otherwise stated, "a plurality of" means two or more.

[0035] The technical solution of the present invention will be further described below with reference to the embodiments and accompanying drawings.

[0036] Example 1 Please see Figure 1 As shown in the figure, this application provides a natural language-based in-vehicle broadcast interaction method, including: S1. Acquire broadcast audio streams, perform broadcast content understanding to obtain structured content data, associate and store the structured content data with corresponding timestamps, and construct a broadcast content time index library, including: S11. The broadcast audio stream is collected in parallel between the digital audio output terminal of the car radio tuner and the power amplifier input terminal through a high impedance differential probe. Specifically, during the user's listening to the radio, the original PCM audio stream is obtained by acquiring the radio audio stream synchronized with the program the user is currently listening to in real time.

[0037] S12. Timestamp synchronization: Add a synchronization frame containing the current time information to the broadcast audio stream to obtain the target audio stream; Specifically, the system has a built-in independent hardware counter (with an accuracy of 1ms) that inserts a synchronization frame every 10~50ms, inserting a synchronization frame containing the current time information into the broadcast audio stream.

[0038] S13. Perform multimodal content recognition on the target audio stream and detect broadcast frequency switching events in real time to preserve the context, and then integrate to obtain structured content data; In this embodiment, multimodal content recognition of the target audio stream includes: Based on streaming speech recognition, the target audio stream is converted into ASR transcribed text, and the start and end timestamps of each sentence are recorded. Based on music fingerprinting, the metadata of songs in the target audio stream is identified, and the timestamp of the identification time is recorded; the metadata includes data such as song title and artist; The target audio stream is structured using speaker log technology, the speaker audio is segmented, and it is categorized based on role tags.

[0039] Specifically, voiceprint recognition or speaker clustering algorithms are used to distinguish different roles such as host and guests, and each audio segment is labeled with a role tag (such as "host", "guest A", "guest B") and associated with a timestamp. Among them, role tags are identified by a pre-set host voiceprint model, or different speakers are temporarily distinguished by unsupervised clustering when no pre-set information is available.

[0040] This embodiment applies multimodal content recognition technology (streaming speech recognition, music fingerprinting, speaker logs, etc.) to the target audio stream, transforming unstructured audio signals into structured, searchable, and semantically rich structured content data in real time. This enables multi-dimensional content parsing and understanding, allowing for simultaneous understanding of audio content from different dimensions. This supports efficient retrieval, accurate content positioning, and precise response to user language commands, thus meeting user interaction needs.

[0041] In this embodiment, real-time detection of broadcast frequency switching events to preserve context includes: A1. When a broadcast frequency switching event is detected, the target audio segment is obtained from the broadcast audio stream before the switching, based on the system time at the time of the switching. The detection method for broadcast frequency switching events can be selected according to requirements, as follows: (1) RDS signal detection: monitoring frequency information changes in the wireless data system (RDS); (2) Silence detection: detects brief silences in the audio stream, which are usually accompanied by brief silences when the broadcast frequency is switched; (3) Tuner status monitoring: Directly read the current frequency register value of the car radio tuner; (4) User operation monitoring: Monitor whether the user triggers frequency switching through physical buttons or touch screen.

[0042] A2. Extract features from the target audio segment to generate an audio feature vector; Feature extraction schemes include, but are not limited to: MFCC (Mel frequency cepstral coefficients): Extracts the acoustic features of audio. DNN embedding (deep neural network embedding): extracting high-dimensional semantic features through pre-trained models; Music fingerprint: If the audio is a music program, the music fingerprint can be directly extracted for subsequent identification.

[0043] A3. Obtain the audio information and audio feature vector of the target audio segment, and save it as a history record to preserve the context; Specifically, when a frequency switch is detected, the system time Tswitch at the time of the switch and the frequency identifier Fprev before the switch are recorded. Based on the system time Tswitch, a target audio segment of a preset duration (e.g., 5-15 seconds, Δt can be configured according to the scenario) is extracted from the broadcast audio stream before the switch. Feature extraction is performed on the target audio segment to generate an audio feature vector, and a new historical content record is generated. The historical content record contains the following fields.

[0044] time_range_start: The start time of the audio segment before switching (Tswitch) Δt; time_range_end: The end time of the audio segment before switching (Tswitch); frequency: The frequency identifier Fprev before the switch; feature_type: The type of audio feature extracted (e.g., "MFCC"); feature_vector: Extracted audio feature vector data; record_type: Set to "historical" (meaning the record is historical content).

[0045] Insert the historical content record into the broadcast content time index library. The timestamp field uses the original value without any offset or conversion.

[0046] This embodiment determines the switching time of the broadcast frequency switching event based on the system time, then extracts the broadcast audio stream before the switch, obtains the corresponding audio information and audio feature vector, and saves it as a historical content record to retain the context. By retaining the audio features before the switch, it is possible to trace back across frequencies.

[0047] S14. Obtain the synchronization frame corresponding to the structured content data, then determine the corresponding timestamp and associate it with the structured content data for storage, and generate a broadcast content time index library.

[0048] Specifically, all recognition results (text, song ID, role tags, etc.) are associated and stored with their corresponding timestamps to generate a broadcast content time index library. This broadcast content time index library includes one or more of the following: music fingerprint recognition results, ASR transcribed text, speaker log results, and audio feature vectors. Music fingerprint recognition results: stored as a (timestamp, song title, artist) triple; ASR transcribed text: stored as a triplet of (start timestamp, end timestamp, text fragment); Speaker log results: stored as a triplet of (start timestamp, end timestamp, role tag); Audio feature vector: Stored as an index record containing the original time interval [Tswitch-Δt, Tswitch], frequency identifier, feature type, feature vector, and record type marker "historical".

[0049] For example: In the music collection scenario, the system identifies the song "Ru Yuan" at timestamp T1 and stores (T1, song title: "Ru Yuan", singer: Faye Wong) in the index.

[0050] In news recap scenarios, the system stores the ASR transcribed text by timestamp: (T2_start, T2_end, "The new energy vehicle purchase subsidy policy will continue until the end of 2025, but the subsidy standard will be reduced by 30% compared to 2023").

[0051] In the station switching memory scenario, the system detects a switch from FM 97.4 to FM 88.7, immediately saves the audio MFCC feature vector of the 10 seconds prior to the switch, and generates an index record containing the original time interval [Tswitch-10s, Tswitch], frequency identifier FM97.4, feature type MFCC, feature vector data, and record type "historical".

[0052] In the role-referencing scenario, the system labels each voice segment with a role tag and stores them as (T3_start, T3_end, "Host"), (T4_start, T4_end, "Guest"), etc.

[0053] This embodiment is based on audio bypass acquisition. A high-impedance differential probe is used to acquire the broadcast audio stream in parallel between the digital audio output of the car radio tuner and the input of the power amplifier. This allows for the acquisition of audio completely synchronized with the user with zero latency and zero sound loss without interrupting the original playback path. A synchronization frame containing current time information is added to the broadcast audio stream to obtain the target audio stream. Hardware timestamp synchronization ensures that each audio segment has a unique and accurate time identifier, allowing users to accurately query and rewind audio as needed.

[0054] S2. Obtain user language commands, perform speech recognition and natural language understanding, parse user intent and time indication information, and then determine the backtracking time window based on the time indication information, including: Collect user voice and recognize it to obtain user language commands; Natural language understanding is used to interpret user commands and extract user intent and time indication information. The time indication information is semantically parsed to identify the specific time direction and range boundary, and the corresponding backtracking time window is dynamically calculated.

[0055] In this embodiment, the user triggers voice interaction via a voice wake-up word (such as "Hello, Xiao X") or a physical button (such as the voice button on the steering wheel). The system collects the user's voice and converts it into text commands (i.e., user language commands) through speech recognition, or directly performs semantic understanding on the speech.

[0056] For example, users can say natural language commands such as "Add this song to your favorites", "How much has the subsidy been reduced?", "What was the name of that song?", or "What is the L3 standard that the host just mentioned?"

[0057] Intent and reference parsing: Through the Natural Language Understanding (NLU) module, user intent (such as "favorite", "search", "play" etc.) and time indication information (such as "current", "just now", "previous song", "host said" etc.) are identified.

[0058] In this embodiment, the specific mapping rules for the backtracking time window are as follows: 1. If the time indication information is "current", then the backtracking time window is the time point closest to the instruction reception time (i.e., a short period of time near the current time, such as within ±1 second). 2. If the time indicator is "just now", the time window will be the time period preceding the current time by the first preset duration (e.g., 5 minutes in the past). This duration can be dynamically adjusted according to the context. If it is detected that the user has switched frequencies within the most recent preset time period (e.g., 10 seconds), then the following steps are performed: Get the time of the most recent switch, Tswitch, and the time interval of the historical content retained before the switch, [Tswitch-Δt, Tswitch]. Calculate the initial time window: Winitial=[Tnow [L,Tnow], where Tnow is the system time at the time of the query, and L is the base duration (e.g., 300 seconds). If the switching time is within the time window, i.e., Tswitch∈Winitial, then the window is expanded to Wextended=[min(Tswitch)]. Δt,Tnow [L),Tnow] (Select the revelation node with the shorter time window from the initial node and the historical content retention starting point); otherwise, keep the initial window unchanged, i.e., Wextended=Winitial; The final backtracking window is determined to be Wextended.

[0059] 3. If the time indicator is "Previous Track", then the backtracking time window is the start and end times of the previous music unit. The boundaries of a music unit can be determined in the following ways: ① Audio energy detection (silent segments between songs); ② Changes in the continuous matching results of music fingerprints; ③Program type switching indication in broadcast RDS information.

[0060] Thus, the start and end timestamp range of the previous song is determined based on the music unit.

[0061] 4. If the time indication information is "said by the host", then the backtracking time window is the audio range of the most recent speaker labeled "host"; 5. Other time indicators such as "5 minutes ago" or "that news clip" can be mapped to the corresponding time period.

[0062] For example, in the command "Add this song to your favorites", the intent is parsed as "add to favorites" and the time indicator is "current". In the command "How much has the subsidy been reduced?", "just now" is parsed as a time window W = [Tnow - 300s, Tnow]. In the channel switching recall scenario, when a user makes a query on a new channel, the system detects that a frequency switch has recently occurred. Based on the switch time Tswitch and the retained historical interval [Tswitch - 10s, Tswitch], the initial window [Tnow - 300s, Tnow] is expanded to [min(Tswitch - 10s, Tnow - 300s), Tnow], thereby retrieving content before and after the switch simultaneously. In the role referencing scenario, the command is parsed as a time reference to "just now" and a role reference to "host". The system locates the audio time segment where the speaker is labeled "host".

[0063] This embodiment collects the user's natural language commands, performs speech recognition and natural language understanding, and parses out the user's intent and time indication information. In this way, interactive operation can be achieved without manual operation. Voice interaction control effectively avoids the risk of blind driving caused by distracted screen operation, and greatly improves driving safety.

[0064] S3. Using the aforementioned retrospective time window as an index, retrieve the corresponding structured content data from the broadcast content time index library, including: Using the start and end time nodes of the backtracking time window as the retrieval boundary, the broadcast content time index library is quickly retrieved to obtain all structured content data within this time period. Traverse all structured content data, perform semantic similarity calculation on the ASR transcribed text based on keyword matching or using a vector model, and select and output the structured content data with the highest content matching degree with the user's language instruction.

[0065] Specifically: First, perform data retrieval: Based on the retrospective time window, quickly retrieve all structured content data stored within that time period from the broadcast content time index library, including song metadata, ASR transcribed text fragments, audio features, character tags, etc.

[0066] Secondly, perform content localization: After retrieving structured content data from the broadcast content time index library, if more precise localization is needed (e.g., when multiple ASR transcribed texts or multiple songs are contained within the time window, and further content localization is required), keyword matching or semantic search can be performed on the ASR transcribed texts within the time window.

[0067] If the structured content data retrieved based on the backtracking time window is empty, then keyword matching or semantic search is performed on the ASR transcribed text within the backtracking time window to locate the most relevant content fragments. The keywords are derived from the core words in the user command excluding time references, such as "subsidy" and "reduction" in the command "How much did the subsidy you just mentioned decrease?"

[0068] The specific search scenarios are as follows: In the music collection scenario, obtain the current system time T_now, retrieve the song record with the timestamp closest to T_now from the index, and extract the metadata of "Ru Yuan".

[0069] In the news recap scenario, the ASR transcribed text within the recap time window [T_now-300s, T_now] is located by keyword matching (such as "subsidy" and "phase-out") to find sentences containing "phase-out 30%".

[0070] In the scene of switching stages and recalling memories, the audio feature vectors preserved before the stage switch are sent to the music fingerprint database to match the song "As You Wish".

[0071] In the context of character reference, the audio time segment of the most recent host character is located, the ASR transcribed text of that segment is extracted, and the entity "L3 autonomous driving" is extracted from it.

[0072] This embodiment uses the start and end time nodes of the retrospective time window as the retrieval boundary to quickly search the broadcast content time index library and achieve millisecond-level timestamp positioning; and by combining keyword matching and semantic similarity calculation, the structured content data with the highest content matching degree can be retrieved to accurately locate the content indicated by the user.

[0073] S4. Based on the user intent, process the structured content data to perform the corresponding target operation and / or generate response content and perform feedback.

[0074] In this embodiment, the execution of the target operation includes: If the user's intention is to collect songs, extract the song metadata from the structured content data and write it into the user's personal playlist database; If the user's intent is to query content, the answer text is generated by calling the local knowledge base and combining the structured content data, with the keywords in the user's language command as the core. If the user's intent is to play audio, then the corresponding audio segment is extracted from the structured content data and pushed to the player.

[0075] The local knowledge base includes pre-built encyclopedia entries, explanations of technical terms, and historical question-and-answer records. This local knowledge base is independent of the broadcast content index and is used to provide expanded explanations of entities mentioned in the broadcast.

[0076] This embodiment uses structured content data obtained from broadcast audio streams to seamlessly translate natural language commands into specific backend business actions, thereby improving interaction efficiency and user experience.

[0077] In this embodiment, the feedback mode includes one or more of the following: voice broadcast, tactile vibration, graphic card display, highlighted original text display, playlist, and progress bar anchor point display. Specifically: a. Voice broadcast: Suitable for brief confirmations or answers, supporting user interruption, replay, and speech rate adjustment; b. Image and text card display: The song cover, artist, news summary, etc. are displayed in a non-intrusive card format on the edge area of ​​the central control screen, and automatically fade out after a preset time; c. Highlighting the original text: In the ASR transcription text view or the full-text news interface, the original sentence corresponding to the answer is highlighted with a background color, and click to listen back; d. Playlist: Generates structured playlists in chronological order, supporting export to mobile app or USB; e. Progress bar anchor points: Generate visual anchor point markers on the audio progress bar, allowing users to directly listen back to the corresponding segments via voice or touch. f. Tactile vibration: Provides operational confirmation through vibration of the steering wheel or seat.

[0078] This embodiment provides multimodal adaptive feedback, with voice broadcasting, tactile vibration, graphic card display, highlighted original text display, progress bar anchor point display, and other multi-directional collaboration, effectively improving the efficiency and accuracy of information reception and meeting personalization and scene adaptation requirements.

[0079] In this embodiment, the execution feedback includes: Obtain the current driving status and identify the vehicle's driving load; Pre-map one-to-one corresponding driving load levels and feedback modes, and then match the corresponding feedback modes according to the driving load; Feedback priorities are determined based on the urgency of the target action / response content, and then feedback is executed based on the feedback pattern and user historical preferences.

[0080] Specifically, in this embodiment, the optimal feedback combination is dynamically selected by combining multiple dimensions such as information urgency, driving load, user historical preferences, and in-vehicle screen status. This combination can be used individually or in combination to execute the feedback. Among these: Information urgency: For example, prompts related to driving safety (such as sudden changes in road conditions) use voice + vibration for high-priority feedback, while music collection confirmation can be done with just a card + voice. Driving load: The current driving attention requirement is determined by the current driving status. When the load is high, visual feedback is reduced and voice and tactile feedback are enhanced. The current driving status includes information such as vehicle speed, steering wheel angle, and road conditions. User history preferences: Records users' acceptance of feedback formats, such as whether they frequently close graphic cards; In-vehicle screen status: When the screen is detected to be off, only voice and vibration feedback are used.

[0081] For example: In the music collection scenario, the system performs the collection operation and announces "The current song 'As You Wish' by Faye Wong has been collected for you" via voice, while simultaneously displaying a graphic card for dual feedback.

[0082] In the news recap scenario, the system's voice response is "The Ministry of Finance announced that the new energy vehicle purchase subsidy standard will be reduced by 30% compared to 2023," while highlighting the original sentence on the news summary interface and providing an access point for listening back.

[0083] In the station recall scene, the system voice replies, "The song you just listened to on FM 97.4 is 'As You Wish,' sung by Faye Wong," and displays a song information card with a "Favorite" button.

[0084] In the role-referencing scenario, the system provides a voice response with the L3 standard definition and generates anchor marks on the progress bar, allowing users to jump back and listen again.

[0085] This embodiment identifies the vehicle's driving load in real time by acquiring the current driving status. In high-load scenarios, it automatically suppresses complex visual feedback and switches to concise voice commands to avoid distracting the driver. In low-load scenarios, it provides rich and intuitive multimodal feedback to improve the interactive experience. By combining the urgency of information with feedback priority, it ensures that critical information (mainly related to safe driving) is delivered to the driver promptly, while non-urgent entertainment information is weakened or delayed, reducing cognitive overload and optimizing interaction efficiency. Multimodal feedback is dynamically executed based on the user's historical preferences (such as preference for voice or dashboard prompts), making the interaction more natural and achieving a personalized experience.

[0086] This invention collects broadcast audio streams, performs broadcast content understanding to obtain structured content data, associates and stores the structured content data with corresponding timestamps, constructs a broadcast content time index library, and transforms unstructured audio streams into structured data with timestamps. When a user's verbal command is detected, speech recognition and natural language understanding are performed to parse the user's intent and time indication information, thereby determining a backtracking time window and quickly retrieving the corresponding structured content data using the backtracking time window as an index. Finally, based on the user's intent, the structured content data is processed to execute corresponding target operations and / or generate response content and perform multimodal feedback. This enables interactive operations such as replay, query, or information extraction (i.e., supports rich interactive functions such as music collection, news playback, channel switching recall, and role identification), transforming broadcasting from a one-way receiving device into a conversational intelligent terminal. The elimination of manual operation significantly reduces the risk of driver distraction, improves driving safety and information acquisition efficiency, and provides users with a good in-vehicle radio listening experience.

[0087] Example 2 See Figure 2 The figures in the accompanying drawings of this embodiment include: time synchronization module 1, audio acquisition module 2, content recognition module 3, index storage module 4, voice interaction module 5, backtracking calculation module 6, retrieval execution module 7, and feedback module 8.

[0088] This invention also provides a natural language-based vehicle broadcasting interaction system for implementing the natural language-based vehicle broadcasting interaction method described in Embodiment 1 above. The system includes a time synchronization module 1, an audio acquisition module 2, a content recognition module 3, and an index storage module 4 connected in sequence, as well as a voice interaction module 5, a backtracking calculation module 6, and a retrieval execution module 7 connected in sequence. It also includes a feedback module 8 connected to the retrieval execution module 7, and the index storage module 4 is further connected to the retrieval execution module 7. The time synchronization module 1 is used to perform timestamp synchronization, has a built-in hardware counter, and outputs a synchronization frame including the current time information. The audio acquisition module 2 is used to acquire broadcast audio streams and add synchronization frames to output the target audio stream; The content recognition module 3 is used to perform broadcast content understanding on the target audio stream to obtain structured content data; The index storage module 4 is used to associate and store the structured content data with the corresponding timestamps to build a broadcast content time index library to support efficient time range retrieval. The voice interaction module 5 is used to acquire user language commands, perform voice recognition and natural language understanding, and parse user intent and time indication information. The backtracking calculation module 6 is used to determine the backtracking time window based on the time indication information, and supports multiple semantic mappings (current, just now, previous song, role reference, etc.). The retrieval execution module 7 is used to retrieve the corresponding structured content data from the broadcast content time index library using the backtracking time window as an index, and to process the structured content data based on the user intent in order to perform the corresponding target operation and / or generate response content. The feedback module 8 is used to perform feedback and provide results to the user. It supports various feedback formats such as voice broadcast, graphic cards, highlighted text, playlists, progress bar anchors, and tactile vibration, and can be dynamically combined according to the scenario.

[0089] In this embodiment, the core functions are processed offline by the vehicle to protect privacy and are fully available when there is no network. If the device is connected to the network, it can further call the cloud knowledge base or search engine to obtain richer extended information (such as entity encyclopedia explanations), but this is not an essential function.

[0090] This invention constructs a natural language-based in-vehicle broadcast interaction system, providing a foundation for broadcast interaction. This allows users to respond instantly, accurately trace, and manage broadcast content through everyday conversational commands (such as "add this song to your favorites," "how much the subsidy reduction was mentioned earlier," and "the standards mentioned by the host"). The entire process requires no manual operation, no application switching, and no reliance on the network, making broadcasting an "interactive" part of the smart cockpit.

[0091] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A natural language-based in-vehicle broadcasting interaction method, characterized in that, include: The broadcast audio stream is collected, the broadcast content is understood to obtain structured content data, the structured content data is associated with the corresponding timestamps and stored, and a broadcast content time index library is built. The system acquires user language commands, performs speech recognition and natural language understanding, parses user intent and time indication information, and then determines the backtracking time window based on the time indication information. Using the aforementioned retrospective time window as an index, the corresponding structured content data is retrieved from the broadcast content time index library; Based on the user's intent, the structured content data is processed to perform the corresponding target operation and / or generate response content and perform feedback.

2. The in-vehicle broadcast interaction method based on natural language as described in claim 1, characterized in that, The system acquires broadcast audio streams, performs broadcast content understanding to obtain structured content data, associates and stores the structured content data with corresponding timestamps, and constructs a broadcast content time index library, including: The broadcast audio stream is collected in parallel between the digital audio output terminal of the car radio tuner and the power amplifier input terminal through a high-impedance differential probe; Timestamp synchronization involves adding a synchronization frame containing the current time information to the broadcast audio stream to obtain the target audio stream. Multimodal content recognition is performed on the target audio stream, and broadcast frequency switching events are detected in real time to preserve context, thereby integrating the data to obtain structured content data. Obtain the synchronization frame corresponding to the structured content data, then determine the corresponding timestamp and associate it with the structured content data for storage, generating a broadcast content time index library.

3. The in-vehicle broadcast interaction method based on natural language as described in claim 2, characterized in that, Multimodal content recognition of the target audio stream includes: Based on streaming speech recognition, the target audio stream is converted into ASR transcribed text, and the start and end timestamps of each sentence are recorded. Based on music fingerprint recognition, the metadata of songs in the target audio stream is identified, and the timestamp of the identification time is recorded; The target audio stream is structured using speaker log technology, the speaker audio is segmented, and it is categorized based on role tags.

4. The in-vehicle broadcast interaction method based on natural language as described in claim 3, characterized in that, Real-time detection of broadcast frequency switching events to preserve context, including: When a broadcast frequency switching event is detected, the target audio segment is obtained from the broadcast audio stream before the switching, based on the system time at the time of the switching. Feature extraction is performed on the target audio segment to generate an audio feature vector; Obtain the audio information and audio feature vector of the target audio segment, and save it as a history record to preserve the context; The broadcast content time index library includes one or more of the following: music fingerprint recognition results, ASR transcribed text, speaker log results, and audio feature vectors.

5. The in-vehicle broadcast interaction method based on natural language as described in claim 1, characterized in that, The process involves acquiring user language commands, performing speech recognition and natural language understanding, parsing user intent and time indication information, and then determining a backtracking time window based on the time indication information, including: Collect user voice and recognize it to obtain user language commands; Natural language understanding is used to interpret user commands and extract user intent and time indication information. The time indication information is semantically parsed to identify the specific time direction and range boundary, and the corresponding backtracking time window is dynamically calculated.

6. The in-vehicle broadcast interaction method based on natural language as described in claim 1, characterized in that, Using the aforementioned retrospective time window as an index, the corresponding structured content data is retrieved from the broadcast content time index library, including: Using the start and end time nodes of the backtracking time window as the retrieval boundary, the broadcast content time index library is quickly retrieved to obtain all structured content data within this time period. Traverse all structured content data, perform semantic similarity calculation on the ASR transcribed text based on keyword matching or using a vector model, and select and output the structured content data with the highest content matching degree with the user's language instruction.

7. The in-vehicle broadcast interaction method based on natural language as described in claim 1, characterized in that, Execution feedback includes: Obtain the current driving status and identify the vehicle's driving load; Pre-map one-to-one corresponding driving load levels and feedback modes, and then match the corresponding feedback modes according to the driving load; Feedback priorities are determined based on the urgency of the target action / response content, and then feedback is executed based on the feedback pattern and user historical preferences.

8. The in-vehicle broadcast interaction method based on natural language as described in claim 7, characterized in that, The execution of the target operation includes: If the user's intention is to collect songs, extract the song metadata from the structured content data and write it into the user's personal playlist database; If the user's intent is to query content, the answer text is generated by calling the local knowledge base and combining the structured content data, with the keywords in the user's language command as the core. If the user's intent is to play audio, then the corresponding audio segment is extracted from the structured content data and pushed to the player.

9. The in-vehicle broadcast interaction method based on natural language as described in claim 7, characterized in that: Feedback modes include one or more of the following: voice broadcast, tactile vibration, graphic card display, highlighted original text display, playlist, and progress bar anchor point display.

10. A natural language-based in-vehicle broadcasting interaction system, used to implement the natural language-based in-vehicle broadcasting interaction method as described in any one of claims 1 to 9, characterized in that: It includes a time synchronization module, an audio acquisition module, a content recognition module, and an index storage module connected in sequence, as well as a voice interaction module, a backtracking calculation module, and a retrieval execution module connected in sequence. It also includes a feedback module connected to the retrieval execution module, and the index storage module is also connected to the retrieval execution module. The time synchronization module is used to perform timestamp synchronization and output a synchronization frame including the current time information; The audio acquisition module is used to acquire broadcast audio streams and add synchronization frames to output the target audio stream; The content recognition module is used to perform broadcast content understanding on the target audio stream to obtain structured content data; The index storage module is used to associate and store the structured content data with the corresponding timestamps to build a broadcast content time index library; The voice interaction module is used to acquire user language commands, perform speech recognition and natural language understanding, and parse user intent and time indication information; The backtracking calculation module is used to determine the backtracking time window based on the time indication information; The retrieval execution module is used to retrieve the corresponding structured content data from the broadcast content time index library using the backtracking time window as an index, and to process the structured content data based on the user intent in order to perform the corresponding target operation and / or generate response content. The feedback module is used to perform feedback.

Images