Speech recognition method, device, mobile device and computer readable storage medium

Abstract

Embodiments of the present application disclose a speech recognition method and device, a mobile device and a computer readable storage medium. The method is applied to a mobile device, and comprises the following steps: determining at least one target sound area located in the internal space of the mobile device and second audio information corresponding to each target sound area in the at least one target sound area based on first audio information collected by the mobile device; the internal space of the mobile device is divided into multiple sound areas, and different sound areas correspond to different subspaces in the internal space; for each target sound area, determining a first recognition result of the target sound area based on the second audio information and the first video information corresponding to the target sound area. In this way, the accuracy and reliability of speech recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of automatic speech recognition technology, and includes, but is not limited to, a speech recognition method, apparatus, mobile device, and computer-readable storage medium. Background Technology

[0002] While mobile devices are in motion, voice recognition can control them without interfering with driving, and it can also upgrade interaction from mechanical commands to natural dialogue, thereby improving the safety and user experience of mobile devices. However, in related technologies, voice recognition results are still affected by various interference sources, such as the voices of other passengers, audio from electronic devices, prompts from mobile devices, and ambient reflections. Therefore, voice recognition suffers from low accuracy and poor stability. Summary of the Invention

[0003] In view of this, the speech recognition method, apparatus, and computer-readable storage medium of the mobile device provided in the embodiments of this application can improve the accuracy and reliability of speech recognition. The speech recognition method, apparatus, mobile device, computer-readable storage medium, and computer program product provided in the embodiments of this application are implemented as follows: A first aspect of this application provides a speech recognition method applied to a mobile device, the method comprising: Based on the first audio information collected by the mobile device, at least one target sound zone located in the internal space of the mobile device is determined, and second audio information corresponding to each target sound zone in the at least one target sound zone is determined; the internal space of the mobile device is divided into multiple sound zones, and different sound zones correspond to different subspaces in the internal space; For each target audio region, a first recognition result for the target audio region is determined based on the second audio information and the first video information corresponding to the target audio region.

[0004] The speech recognition method provided in this application embodiment determines the first recognition result based on information from these two different modalities: the second audio information belongs to the audio modality, and the first video information belongs to the video modality. This means that speech recognition is performed using multimodal information, thereby improving the accuracy of speech recognition. Furthermore, this application embodiment first filters out the target audio region containing the second audio information, and then performs multimodal speech recognition on the target audio region, without needing to perform multimodal speech recognition on non-target audio regions that do not contain the second audio information. This simplifies the speech recognition process, reduces computational complexity, and improves the reliability and efficiency of speech recognition.

[0005] In some embodiments, the first video information includes multiple video frames, and each video frame includes at least one candidate object; determining the first recognition result of the target audio region based on the second audio information and the first video information corresponding to the target audio region includes: performing vocal recognition on each candidate object of the at least one candidate object based on the multiple video frames, and determining the candidate object in a vocal state as the target object; determining the audio information in the second audio information corresponding to the target object as the first recognition result of the target audio region.

[0006] The aforementioned multimodal recognition process achieves accurate sound source separation and directional identification in the complex internal environment of mobile devices through joint modeling of "visual" and "auditory" senses. Utilizing video frames for lip movement or facial motion detection effectively distinguishes who is actually speaking, thus accurately locating the target speaker even with multiple people speaking simultaneously or strong background noise (such as tire noise or music), demonstrating strong anti-interference capabilities and improving the robustness of audio separation. Using "vocalization status" as a hard screening condition avoids potential locational misjudgments that may occur when relying solely on audio beamforming, ensuring that the final first recognition result only includes the voice of the actual speaker in the target vocal range, achieving accurate interaction and reducing false triggers from irrelevant voices from the passenger or rear seats. Visual information provides temporally synchronized "vocalization evidence," assisting the audio processing module in maintaining high recognition accuracy even under extremely low signal-to-noise ratios, achieving audio-video complementarity, and laying the foundation for further advanced functions such as lip-reading enhancement and speaker tracking.

[0007] In some embodiments, each video frame further includes a detection box of the at least one candidate object; the step of performing voice recognition on each candidate object of the at least one candidate object based on the plurality of video frames, and determining the candidate object in a voice state as the target object, includes: for each candidate object among the at least one candidate object, obtaining image information corresponding to the detection box of the candidate object from the plurality of video frames; performing voice recognition based on the image information corresponding to the plurality of detection boxes of the candidate object to obtain a second recognition result of the candidate object; and determining the candidate object as the target object when the second recognition result indicates that the candidate object is in a voice state.

[0008] The aforementioned process of identifying the target object refines visual speech recognition from "global fuzzy judgment" to "individual local temporal verification," improving the accuracy and robustness of target locking. By extracting image information within the detection box, the system can filter out background clutter (such as the body movements of other passengers in the vehicle or changes in the scenery outside the window), forcing visual attention to focus on the mouth and facial area of ​​the target object. This achieves precise anti-interference in the spatial dimension and improves the signal-to-noise ratio of speech features (such as lip movements). Cross-frame continuous recognition is based on "multiple detection box identifiers," rather than relying on a single static image. This effectively avoids single-frame misjudgments caused by momentary non-vocal actions such as "opening the mouth to yawn" or "turning the head," achieving temporal dynamic verification in the temporal dimension and confirming the continuous speech state through dynamic sequences. An independent "acquisition, recognition, and judgment" closed loop is executed for each candidate object, achieving a precise one-to-one mapping between vocal regions and individuals. Even if multiple candidate objects overlap or change positions in the image, the speech state of each object can be tracked independently, providing a strong logical closed loop for ultimately assigning the audio to the unique correct target object.

[0009] In some embodiments, the step of performing voice recognition based on image information corresponding to multiple detection boxes of the candidate object to obtain a second recognition result of the candidate object includes: extracting features from each image information corresponding to the multiple detection boxes to obtain image features; when the image features indicate that a preset part of the candidate object is in motion, determining the identifier of the video frame corresponding to the image information as a motion identifier; obtaining the identifiers of the multiple video frames; and when the number of consecutive identifiers of the multiple video frames that are the motion identifiers is greater than a number threshold, obtaining a second recognition result indicating that the candidate object is in a voice-producing state.

[0010] By employing a dual mechanism of "continuous motion detection" and "threshold determination," speech recognition is upgraded from "single-frame intuition" to "temporal logic verification," improving its resistance to accidental touches and the reliability of its judgments. It does not rely on the instantaneous state of a single image (such as opening the mouth to yawn or turning the head to speak), but rather judges based on the accumulation of motion features across multiple consecutive frames. Effectively filtering out momentary interference, this "continuous motion counting" mechanism can accurately eliminate occasional misjudgments caused by sudden changes in facial expressions or flickering light. This enhances the robustness of speech recognition. It extracts motion features from preset areas (such as the mouth region) rather than the high-dimensional details of the entire face, reducing the computational load of feature extraction. Simultaneously, the "motion identifier" only records the binary state of "whether there is movement," resulting in minimal memory usage and reduced computational and storage overhead, making it suitable for real-time processing in automotive chips. Introducing "the number of continuous movements exceeding a threshold" as the basis for speech recognition allows developers to dynamically adjust the threshold according to actual scenarios (such as vehicle speed and noise frequency bands), meaning the judgment standard is adjustable and controllable. A higher threshold results in stronger noise resistance; a lower threshold results in higher sensitivity, achieving flexible adaptation between engineering accuracy and application scenarios. This improves the flexibility of speech recognition. The "vocal state" confirmed through this logic has temporal continuity and can serve as high-confidence visual evidence, providing a strong chain of evidence for audio and video attribution and ensuring that subsequent audio attribution will not be incorrectly bound to a non-target speaker due to a single frame deviation.

[0011] In some embodiments, determining the audio information corresponding to the target object in the second audio information as the first recognition result of the target sound region includes: determining multiple video frames identified by the motion identifier as a target video frame sequence; determining the timestamps corresponding to the first and last video frames in the target video frame sequence as the motion timestamps of the target object; and truncating the second audio information based on the motion timestamps to obtain the first recognition result of the target sound region.

[0012] The process of determining the first recognition result described above utilizes visually determined "start and end timestamps" to truncate the audio. This precisely removes all environmental noise, silent segments, and other interference before and after the target speaker begins speaking, achieving accurate alignment and background purity. This ensures that the audio segment fed into the speech recognition engine corresponds exactly to the phonation period, improving input quality. It also enhances the signal-to-noise ratio of speech recognition. By directly replacing the computationally intensive speech activity detection algorithm with visual decision-making, the audio endpoint detection is eliminated, saving computational resources on the automotive chip and reducing truncation errors caused by noise-induced misjudgment of the speech activity detection threshold. The "first video frame" and "last video frame" provide precise time anchors, resolving boundary ambiguity issues caused by reverberation or delay in the audio stream. Clear and unambiguous time boundaries ensure that the final first recognition result corresponds strictly one-to-one with the visual event in terms of time attribution, providing an irrefutable chain of temporal evidence for accurately feeding the recognized text back to the corresponding voice region (e.g., the driver's area).

[0013] In some embodiments, the target object includes a first object and a second object; the step of truncating the second audio information based on the motion timestamp of the target object to obtain a first recognition result of the target audio region includes: truncating the second audio information based on the motion timestamp of the first object to obtain audio information of the first object; truncating the second audio information based on the motion timestamp of the second object to obtain audio information of the second object; the motion timestamp of the first object and the motion timestamp of the second object do not overlap; and determining the audio information of the first object and the audio information of the second object as the first recognition result of the target audio region.

[0014] Through the process of determining the first recognition result described above, precise separation and conflict-free splicing of "multi-person time-sharing speech" are achieved within the same audio region, ensuring that regardless of how many speakers are in the region, each can output pure and complete human-computer interaction content. When the second audio information only contains a single audio region (such as the entire microphone in the back row) but two passengers speak one after the other, this embodiment of the application uses visual timestamps to segment the audio into independent segments by person, effectively distinguishing multiple speakers in the same region, fundamentally solving the problem of not being able to distinguish the speaker's affiliation when "multiple people share one audio stream". Since the motion timestamps of the two do not overlap, it indicates that their respective speaking periods are naturally staggered on the time axis. The non-overlapping time ensures the feasibility of physical truncation, which provides a solid basis for pure audio segmentation, avoids complex algorithms for handling speech aliasing (two people speaking at the same time), and makes the truncation operation inherently conflict-free. The truncated first object audio and second object audio are output as a whole result according to their time sequence, preserving the complete semantic order. This not only preserves their independent and clear pronunciation segments, but also fully presents the logical order of the dialogue within the audio region, providing structured temporal data for subsequent speech recognition or semantic understanding. It can be extended to "first object, second object... Nth object", exhibiting good generalization ability. Furthermore, the audio and video evidence of each speaker is independent; even if a brief anomaly occurs in the feature extraction of one path, it will not affect the normal truncation processing of other objects. It supports dynamic expansion of speech recognition and improves the robustness of speech recognition.

[0015] In some embodiments, before determining the first recognition result of the target audio region based on the second audio information and the first video information corresponding to the target audio region, the method further includes: acquiring second video information that is in the same time period as the first audio information; each video frame in the second video information includes at least one detection box of an object to be identified; performing audio region mapping on the second video information to obtain first video information corresponding to the target audio region; each video frame in the first video information includes at least one detection box of a candidate object, the candidate object being an object to be identified located in the target audio region.

[0016] Through the aforementioned alignment process between audio and video regions, "spatial dimensionality reduction" and "precise isolation" of visual features are achieved at the "audio region" level, ensuring that the input data for subsequent audio-video fusion processing is strictly aligned in physical space. By mapping audio regions, all object detection boxes belonging to other audio regions (such as the passenger side or the left rear seat) in the global video are filtered out. This effectively avoids cross-regional visual interference caused by the passenger's facial movements when the driver is speaking, ensuring that visual features only serve the correct physical region. Compared to performing full object detection and voice analysis on every frame of the global video, the mapped first video only contains candidate objects within the target audio region. This reduces the computational load and memory consumption required for subsequent "object-by-object voice recognition," reducing computational resources and providing the possibility of real-time processing for low-performance automotive chips. It also ensures that the "second audio information" (focused on the audio region by microphone array beamforming) and the "first video information" (mapped from the camera image) describe the same three-dimensional region in physical space. This rigid alignment of "seeing only what you hear" eliminates the potential for spatial misalignment caused by wide-angle distortion of the camera or microphone pickup delay, laying a solid spatiotemporal foundation for subsequent accurate "audio-visual attribution determination".

[0017] In some embodiments, the first audio information is obtained based on the microphone array of the mobile device; determining at least one target sound region located in the internal space of the mobile device, and second audio information corresponding to each target sound region, based on the first audio information collected by the mobile device, includes: preprocessing the first audio information to obtain processed audio information; performing sound source localization on the processed audio information to obtain the location information of the processed audio information; performing sound region mapping on the location information to obtain the at least one target sound region; and extracting the second audio information corresponding to each target sound region from the first audio information.

[0018] Through the aforementioned process of determining the target sound zone level second audio information, the physical advantages of the microphone array are utilized to transform the original multi-channel audio into a structured audio stream "divided by spatial region," achieving a precise mapping between the acoustic space and the physical cabin. Preprocessing operations such as noise reduction, dereverberation, and echo cancellation effectively suppress tire noise, wind noise, and multimedia echoes within the vehicle, improving audio signal quality and providing a clean input with a high signal-to-noise ratio for subsequent sound source localization and sound zone extraction, ensuring accurate localization from the source. Sound source localization algorithms (such as time difference of arrival and beamforming) are used to calculate the sound source location, and then the physical angle is mapped to semantic sound zones such as "driver's seat, passenger's seat, rear left / right." This two-step "localization and mapping" strategy allows for flexible decoupling of the microphone array's hardware capabilities from the cabin's functional domain, enabling the same array to adapt to different vehicle seating layouts. The second audio information for each target sound zone is precisely separated from the original first audio information, preparing a private audio channel for subsequent independent "visual-auditory" fusion recognition of each sound zone. This allows voice commands from different positions, such as the driver and passenger, to be processed in parallel without interference, which is the key foundation for achieving "independent multi-zone interaction throughout the vehicle".

[0019] In some embodiments, the first recognition result includes target audio information; after determining the first recognition result of the target audio region based on the second audio information and the first video information corresponding to the target audio region, the method further includes: determining an operation instruction corresponding to the target audio information; and executing the operation instruction if the operation type of the operation instruction belongs to a preset type set corresponding to the target audio region.

[0020] By introducing a voice zone-based "permission control" and "function isolation" mechanism, voice interaction is upgraded from "full-domain execution" to "zone-based fine-grained control," improving safety, accuracy, and user experience. By strictly limiting the execution permissions of different seats through preset type sets (e.g., the driver can execute high-priority commands such as navigation and driving mode switching, while the front passenger or rear passengers are prohibited from doing so), the risk of misoperation is avoided, fundamentally preventing children or front passenger passengers from accidentally triggering dangerous commands and ensuring driving safety. It ensures that recognized operation commands are accurately applied to the issuing voice zone (e.g., a rear passenger saying "turn up the temperature" only adjusts the rear air conditioning zone, and a front passenger saying "turn on the seat massage" only activates the front passenger seat), providing a personalized experience that fits the zone's individual needs, avoiding accidental control of the entire vehicle or interference with the driver's seat, and meeting the intuitive expectations of each passenger. Before executing an operation command, it quickly verifies whether the operation type belongs to the "whitelist" of that voice zone, effectively filtering out requests that the location is not authorized to make (e.g., a rear passenger attempting to cancel navigation routes). This reduces the computational burden on subsequent execution modules of the system and avoids interactive confusion caused by frequent rejections or erroneous executions, making human-vehicle dialogue smoother and more compliant with rules, thus improving the user experience.

[0021] A second aspect of the embodiments of this application also provides a voice recognition device, the device comprising: a first determining module and a second determining module; The first determining module is used to determine at least one target sound zone located in the internal space of the mobile device based on the first audio information collected by the mobile device, and the second audio information corresponding to each target sound zone in the at least one target sound zone; the internal space of the mobile device is divided into multiple sound zones, and different sound zones correspond to different subspaces in the internal space; The second determining module is used to determine a first recognition result of each target sound region based on the second audio information and the first video information corresponding to the target sound region.

[0022] A third aspect of this application provides a mobile device, which includes a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the speech recognition method of this application.

[0023] In a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the speech recognition method provided in the embodiments of this application.

[0024] A fifth aspect of the embodiments of this application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the speech recognition method provided in the embodiments of this application. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of an application scenario of the speech recognition system provided in the embodiments of this application; Figure 2 This is a schematic flowchart of a speech recognition method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a process for determining a first identification result provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the process of determining a target object provided in an embodiment of this application; Figure 5 This is a schematic diagram of a process for determining a second identification result provided in an embodiment of this application; Figure 6 This is another flowchart illustrating the determination of the first identification result provided in an embodiment of this application; Figure 7 This is another flowchart illustrating the determination of the first identification result provided in an embodiment of this application; Figure 8 This is another schematic flowchart of the speech recognition method provided in the embodiments of this application; Figure 9 This is a schematic flowchart of an information determination method provided in an embodiment of this application; Figure 10 This is another flowchart illustrating the speech recognition method provided in the embodiments of this application; Figure 11 This is a schematic diagram of the framework structure of a speech recognition system provided in an embodiment of this application; Figure 12 This is a schematic diagram of a speech recognition device provided in an embodiment of this application; Figure 13 This is a schematic diagram of a hardware structure of a mobile device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0029] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0031] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0032] 1) Sound zone: In an in-vehicle voice interaction system, a sound zone refers to an independent acoustic area divided according to the physical space inside the vehicle (usually corresponding to a specific seat).

[0033] A sound zone is a semantic segmentation of the cabin space, typically corresponding to specific seating areas such as "driver's seat," "passenger's seat," "left rear seat," and "right rear seat." Sound zones form the spatial coordinate basis for the system to determine "who is speaking."

[0034] The sound zone is not a physical partition, but a virtual area "calculated" using beamforming technology of the microphone array. The system calculates the time difference and phase difference of sound reaching different microphones through sound source localization algorithms (such as time difference of arrival), thereby mapping the sound source location to a specific semantic sound zone (such as mapping the -30° direction to "driver's seat").

[0035] 2) Sound source localization: In the context of in-vehicle voice interaction, sound source localization refers to the technical process of using the sound signals received by the microphone array to calculate the specific location (angle and distance) of the sound source in the three-dimensional space inside the vehicle through algorithms.

[0036] Based on the time difference (arrival time difference), phase difference, or energy difference of sound reaching different microphones, the incident direction of the sound is deduced. Sound source localization is not about "hearing the content," but about "calculating the location." Sound source localization typically outputs specific physical coordinates or angle values ​​(such as "-30° relative to the vehicle's centerline"). Sound source localization provides raw data support for subsequent "sound zone mapping." The system does not directly use the abstract angle "-30°," but instead maps it to a semantic sound zone (such as recognizing this angle as "driver's seat"), thus achieving the interactive basis of "whoever speaks, the system responds to."

[0037] To better understand the speech recognition method provided in the embodiments of this application, the speech recognition methods in related technologies and the technical problems therein will be explained first.

[0038] In in-vehicle voice interaction systems, to improve voice acquisition quality, microphone array technology is typically used to locate sound sources and divide sound zones in different seating areas within the vehicle. This allows candidate voice signals to be acquired from different positions, such as the driver, front passenger, or rear seats. This technology can reduce crosstalk between different seats to some extent, providing a basic input for subsequent voice recognition.

[0039] However, in real-world in-vehicle scenarios, even after audio zone segmentation, multiple interference sources may still exist within the same zone. For example, within a target audio zone, in addition to the target passenger's voice, there may also be voices from other passengers in the same zone, audio from short videos played on mobile phones, navigation announcements, in-vehicle system prompts, and ambient reflected sounds. Because these interferences are spatially close to the target voice, the microphone array-based audio zone segmentation methods of related technologies struggle to further distinguish their sources, potentially leading to misidentification of interference signals as the target passenger's voice.

[0040] Therefore, when there is significant interference in the same audio range within the vehicle, the relevant technologies still struggle to accurately determine which passenger is actually speaking in the current audio range.

[0041] To address the aforementioned problems in related technologies, this application provides a speech recognition method. Since the second audio information belongs to the audio modality and the first video information belongs to the video modality, and these are different modalities, this application determines the first recognition result based on information from these two different modalities. That is, speech recognition is performed using multimodal information, thereby improving the accuracy of speech recognition. Furthermore, this application first filters out the target audio region containing the second audio information, and then performs multimodal speech recognition on the target audio region, without needing to perform multimodal speech recognition on non-target audio regions that do not contain the second audio information. This simplifies the speech recognition process, reduces computational complexity, and improves the reliability and efficiency of speech recognition.

[0042] The following describes exemplary applications of the mobile devices provided in the embodiments of this application. These mobile devices can be implemented as various types of equipment such as automobiles, trains, airplanes, ships, construction machinery, sightseeing vehicles, transport vehicles, autonomous shuttle vehicles, mobile cabins, flying cars, etc. The following will describe exemplary applications when the mobile device is implemented as an automobile.

[0043] Figure 1 This is a schematic diagram illustrating an application scenario of the speech recognition system provided in this application embodiment. Please refer to it. Figure 1 The control system 100 provided in the embodiments of this application can be applied to a vehicle 200. During the driving process of the vehicle 200, the vehicle 200 collects first audio information and second video information inside the vehicle, and performs speech recognition based on the speech recognition method provided in the embodiments of this application. The first audio information and the second video information are two modal information corresponding to the same time period.

[0044] In this embodiment, when the vehicle 200 is in operation, since the second audio information belongs to the audio modality and the first video information belongs to the video modality, and audio and video modalities are different modalities, this embodiment determines the first recognition result based on information from these two different modalities. That is, speech recognition is performed using multimodal information, thereby improving the accuracy of speech recognition. On the other hand, this embodiment first filters out the target audio region containing the second audio information, and then performs multimodal speech recognition on the target audio region, without needing to perform multimodal speech recognition on non-target audio regions that do not contain the second audio information, thereby simplifying the speech recognition process, reducing computational complexity, and improving the reliability and efficiency of speech recognition.

[0045] Figure 2 This is a flowchart illustrating a voice recognition method provided in an embodiment of this application. This voice recognition method can be applied to scenarios such as navigation and travel, driving mode switching, assisted driving adjustment, multimedia control, information inquiry, hands-free communication, basic vehicle control, remote home control, scene linkage, life services, and automatic parking. The executing entity of this voice recognition method can be a mobile device; the following example uses a car as an example. Please refer to... Figure 2 The speech recognition method may include the following steps S201 and S202, which are described in detail below.

[0046] Step S201: Based on the first audio information collected by the mobile device, determine at least one target sound zone located in the internal space of the mobile device, and the second audio information corresponding to each target sound zone in the at least one target sound zone.

[0047] In this embodiment, the internal space of the mobile device is divided into multiple sound zones, and different sound zones correspond to different subspaces within the internal space. Taking a car as an example, the internal space of the car can be divided into a driver's sound zone, a passenger's sound zone, a rear left-side sound zone, and a rear right-side sound zone. The internal space of the car can also be divided into a front sound zone, a middle-side sound zone, and a rear sound zone. These sound zones can be virtual spatial areas calculated using beamforming and sound source localization algorithms from a microphone array; they are not areas obtained through physical partitions. These virtual sound zones can correspond to physical subspaces. For example, the front sound zone corresponds to the subspace where the front seats are located, the middle-side sound zone corresponds to the subspace where the middle seats are located, and the rear sound zone corresponds to the subspace where the rear seats are located.

[0048] In some embodiments, the mobile device may collect first audio information from within the mobile device via its own microphone array. The first audio information may include ambient sounds from outside the mobile device. For example, the first audio information may include the voices of drivers and passengers speaking inside the mobile device.

[0049] In the implementation process, the first audio information undergoes sound source spatial localization, beamforming, and sound propagation delay or intensity difference analysis to distinguish acoustic signals generated by external sound sources and those generated within the mobile device's internal space. Next, based on acoustic characteristics and spatial boundaries, at least one independent target sound region is defined within the mobile device's internal space; each sound region has a corresponding spatial range and acoustic propagation characteristic boundaries. Then, using array signal processing, blind source separation, and spatial filtering, acoustic interference between different target sound regions is extracted from the mixed first audio information—that is, sound region audio decoupling extraction—to obtain the second audio information corresponding to each target sound region.

[0050] Step S202: For each target audio region, determine the first recognition result of the target audio region based on the second audio information and the first video information corresponding to the target audio region.

[0051] In some embodiments, the second audio information corresponding to the target audio region and the first video information of the target audio region can be input into the speech recognition model to obtain the first recognition result of the target audio region. Alternatively, the second audio information and the first video information can be fused first to obtain fused information; then the fused information can be input into the recognition model to obtain the first recognition result of the target audio region. The first recognition result may include at least one of a motion timestamp and audio information.

[0052] In other embodiments, the target object in a vocalizing state can be determined based on the first video information, and the audio information corresponding to the target object in the second audio information can be determined as the first recognition result of the target sound region. The number of target objects can be one or more.

[0053] The speech recognition method provided in this application embodiment determines the first recognition result based on information from these two different modalities: the second audio information belongs to the audio modality, and the first video information belongs to the video modality. This means that speech recognition is performed using multimodal information, thereby improving the accuracy of speech recognition. Furthermore, this application embodiment first filters out the target audio region containing the second audio information, and then performs multimodal speech recognition on the target audio region, without needing to perform multimodal speech recognition on non-target audio regions that do not contain the second audio information. This simplifies the speech recognition process, reduces computational complexity, and improves the reliability and efficiency of speech recognition.

[0054] In some embodiments, the first video information includes multiple video frames, and each video frame includes at least one candidate object. Based on this, Figure 3 This is a flowchart illustrating the determination of a first identification result provided in an embodiment of this application. Please refer to it. Figure 3The above step S202 can be achieved through the following steps S021 and S022, which will be explained in detail below.

[0055] Step S021: Based on multiple video frames, perform voice recognition on each candidate object of at least one candidate object, and determine the candidate object in the voice state as the target object.

[0056] In some embodiments, for each of the at least one candidate object, the candidate object is identified by sound emission based on video frames containing the candidate object in multiple video frames. Candidate objects that are in a sound emission state are identified as target objects, while candidate objects that are not in a sound emission state are identified as non-target objects.

[0057] In one implementation process, a video frame containing each candidate object can be input into the speech recognition model to obtain the speech result for each candidate object. If the speech result indicates that the candidate object is in a speech state, the candidate object is identified as the target object; otherwise, if the speech result indicates that the candidate object is not in a speech state, the candidate object is identified as a non-target object.

[0058] In another implementation, image information of each candidate object can be obtained from the video frame containing each candidate object, and then the image features of the image information can be used to determine whether the candidate object is in a vocalizing state. Finally, if the candidate object is in a vocalizing state, it is identified as the target object.

[0059] Step S022: The audio information in the second audio information that corresponds to the target object is determined as the first recognition result of the target sound region.

[0060] In some embodiments, audio can be extracted from second audio information based on the audio features of the target object to obtain a first recognition result of the target vocal region. The number of target objects can be one or more. Audio features may include at least one of acoustic physical features, prosodic features, speech flow features, and content and pronunciation features.

[0061] In other embodiments, a motion timestamp of the target object is first determined, which characterizes the duration of the target object's vocalization state. Then, the second audio information is truncated based on the motion timestamp to obtain a first recognition result for the target vocal region. For example, assuming the second audio information is a 10-second audio clip and the target object's motion timestamp is 3 to 6 seconds, the audio clips in the 3rd to 6th seconds of the second audio information are determined as the first recognition result for the target vocal region. Alternatively, the motion timestamps of 3 to 6 seconds and the audio clips in the 3rd to 6th seconds of the second audio information can also be determined as the first recognition result for the target vocal region.

[0062] The aforementioned multimodal recognition process achieves accurate sound source separation and directional identification in the complex internal environment of mobile devices through joint modeling of "visual" and "auditory" senses. Utilizing video frames for lip movement or facial motion detection effectively distinguishes who is actually speaking, thus accurately locating the target speaker even with multiple people speaking simultaneously or strong background noise (such as tire noise or music), demonstrating strong anti-interference capabilities and improving the robustness of audio separation. Using "vocalization status" as a hard screening condition avoids potential locational misjudgments that may occur when relying solely on audio beamforming, ensuring that the final first recognition result only includes the voice of the actual speaker in the target vocal range, achieving accurate interaction and reducing false triggers from irrelevant voices from the passenger or rear seats. Visual information provides temporally synchronized "vocalization evidence," assisting the audio processing module in maintaining high recognition accuracy even under extremely low signal-to-noise ratios, achieving audio-video complementarity, and laying the foundation for further advanced functions such as lip-reading enhancement and speaker tracking.

[0063] In some embodiments, each video frame further includes a detection bounding box for at least one candidate object. Based on this, for each candidate object among the at least one candidate object, Figure 4 This is a flowchart illustrating the process of determining a target object according to an embodiment of this application. Please refer to it. Figure 4 The above step S021 can be achieved through the following steps S211 to S213, which will be explained in detail below.

[0064] Step S211: Obtain the image information corresponding to the detection box of the candidate object from multiple video frames.

[0065] In some embodiments, the image information corresponding to the detection box may refer to the image information enclosed by the detection box, or it may refer to the image information of the image area defined by the detection box.

[0066] In the implementation process, image information corresponding to the detection boxes of candidate objects can be extracted from each video frame based on the localization information of the detection boxes. That is, image information of candidate objects can be extracted from video frames, or head image information of candidate objects can be extracted from video frames. The localization information of the detection boxes can be represented using either center point representation or top-left / bottom-right representation.

[0067] Step S212: Based on the image information corresponding to multiple detection boxes of the candidate object, perform voice recognition to obtain the second recognition result of the candidate object.

[0068] In some embodiments, for the image information corresponding to each detection box, feature extraction can be performed on the image information to obtain image features; when any image feature represents a preset part of the candidate object in motion, or when multiple consecutive image features represent preset parts of the candidate objects in motion, a second recognition result is obtained that the candidate object is in a vocalizing state. The preset part may include at least one of the following: mouth, cheek, jaw, throat, etc.

[0069] In other embodiments, when the number of consecutive image features that characterize a preset part of the candidate object in a static state or a preset part of the candidate object in a moving state is less than a number threshold, a second recognition result is obtained in which the candidate object is not in a vocalizing state.

[0070] Step S213: If the second recognition result indicates that the candidate object is in a vocal state, the candidate object is identified as the target object.

[0071] In some embodiments, if a candidate object is in a vocalizing state, it is identified as the target object. If a candidate object is not in a vocalizing state, it is identified as a non-target object.

[0072] The aforementioned process of identifying the target object refines visual speech recognition from "global fuzzy judgment" to "individual local temporal verification," improving the accuracy and robustness of target locking. By extracting image information within the detection box, the system can filter out background clutter (such as the body movements of other passengers in the vehicle or changes in the scenery outside the window), forcing visual attention to focus on the mouth and facial area of ​​the target object. This achieves precise anti-interference in the spatial dimension and improves the signal-to-noise ratio of speech features (such as lip movements). Cross-frame continuous recognition is based on "multiple detection box identifiers," rather than relying on a single static image. This effectively avoids single-frame misjudgments caused by momentary non-vocal actions such as "opening the mouth to yawn" or "turning the head," achieving temporal dynamic verification in the temporal dimension and confirming the continuous speech state through dynamic sequences. An independent "acquisition, recognition, and judgment" closed loop is executed for each candidate object, achieving a precise one-to-one mapping between vocal regions and individuals. Even if multiple candidate objects overlap or change positions in the image, the speech state of each object can be tracked independently, providing a strong logical closed loop for ultimately assigning the audio to the unique correct target object.

[0073] In some embodiments, Figure 5 This is a flowchart illustrating a method for determining a second identification result provided in an embodiment of this application. Please refer to it. Figure 5 For each image information in the image information corresponding to multiple detection boxes, the above step S212 can be implemented by the following steps S121 to S123, which are explained in detail below.

[0074] Step S121: Extract features from the image information to obtain image features.

[0075] In some embodiments, image information can be feature extracted using methods such as histogram of oriented gradients (HARQ) and local binary patterns. Feature extraction can also be performed based on spatial features derived from deep learning. Furthermore, spatiotemporal motion features can be used to extract features from image information.

[0076] Step S122: When the preset part of the candidate object representing the image feature is in motion, the identifier of the video frame corresponding to the image information is determined as the motion identifier; and the identifiers of multiple video frames are obtained.

[0077] In some embodiments, a preset part of the candidate object can reflect whether the candidate object is in a vocal state. The preset part of the candidate object may include at least one of the candidate object's mouth, cheek, jaw, throat, etc.

[0078] In the implementation process, when a preset part of the candidate object representing the image feature is in motion, the corresponding video frame is tagged, and its identifier is determined as a motion identifier. Conversely, when the preset part of the candidate object is stationary, its identifier is determined as a stationary identifier. Based on this, the identifier for each video frame is obtained, thus completing the tagging of each video frame.

[0079] Step S123: If the number of consecutive motion markers in multiple video frames is greater than the number threshold, a second recognition result is obtained that represents the candidate object being in a vocal state.

[0080] In some embodiments, the number of consecutive motion identifiers in multiple video frames is determined. If there are multiple such instances, and the largest of these instances exceeds a threshold, a second recognition result indicating that the candidate object is in a vocalizing state is obtained. For example, suppose the multiple video frames include 10 video frames: video frame 1, video frame 2, video frame 3, video frame 4, video frame 5, video frame 6, video frame 7, video frame 8, video frame 9, and video frame 10. The identifiers of these 10 video frames include motion identifier, motion identifier, still identifier, still identifier, motion identifier, motion identifier, motion identifier, still identifier, still identifier, and so on. Based on this, the number of consecutive motion identifiers in the 10 video frames includes 2 and 4.

[0081] In some embodiments, the quantity threshold is a value pre-set based on experience; for example, the quantity threshold can be 3, 4, 5, etc. Taking a quantity threshold of 3 as an example, continuing the example above, since 4 is greater than 3, a second recognition result representing that the candidate object is in a vocalizing state is obtained.

[0082] In other embodiments, when there are multiple quantities, if the largest quantity among the multiple quantities is less than or equal to a quantity threshold, a second recognition result is obtained that indicates the candidate object is not in a vocal state.

[0083] By employing a dual mechanism of "continuous motion detection" and "threshold determination," speech recognition is upgraded from "single-frame intuition" to "temporal logic verification," improving its resistance to accidental touches and the reliability of its judgments. It does not rely on the instantaneous state of a single image (such as opening the mouth to yawn or turning the head to speak), but rather judges based on the accumulation of motion features across multiple consecutive frames. Effectively filtering out momentary interference, this "continuous motion counting" mechanism can accurately eliminate occasional misjudgments caused by sudden changes in facial expressions or flickering light. This enhances the robustness of speech recognition. It extracts motion features from preset areas (such as the mouth region) rather than the high-dimensional details of the entire face, reducing the computational load of feature extraction. Simultaneously, the "motion identifier" only records the binary state of "whether there is movement," resulting in minimal memory usage and reduced computational and storage overhead, making it suitable for real-time processing in automotive chips. Introducing "the number of continuous movements exceeding a threshold" as the basis for speech recognition allows developers to dynamically adjust the threshold according to actual scenarios (such as vehicle speed and noise frequency bands), meaning the judgment standard is adjustable and controllable. A higher threshold results in stronger noise resistance; a lower threshold results in higher sensitivity, achieving flexible adaptation between engineering accuracy and application scenarios. This improves the flexibility of speech recognition. The "vocal state" confirmed through this logic has temporal continuity and can serve as high-confidence visual evidence, providing a strong chain of evidence for audio and video attribution and ensuring that subsequent audio attribution will not be incorrectly bound to a non-target speaker due to a single frame deviation.

[0084] In some embodiments, Figure 6 This is another flowchart illustrating the determination of the first identification result provided in an embodiment of this application. Please refer to it. Figure 6 The above step S022 can be achieved through the following steps S221 to S223, which will be explained in detail below.

[0085] Step S221: Determine the target video frame sequence from the multiple video frames that are continuously identified as motion markers.

[0086] In some embodiments, following the example above, video frames 5, 6, 7, and 8 are determined as the target video frame sequence.

[0087] Step S222: Determine the timestamps corresponding to the first and last video frames in the target video frame sequence as the motion timestamps of the target object.

[0088] In some embodiments, following the example above, video frame 5 is the first video frame in the target video frame sequence, and video frame 8 is the last video frame in the target video frame sequence. Assuming that the timestamp corresponding to video frame 5 is the 5th second and the timestamp corresponding to video frame 8 is the 8th second, then the timestamps from the 5th to the 8th second are determined as the motion timestamps of the target object.

[0089] Step S223: The second audio information is truncated based on the motion timestamp to obtain the first recognition result of the target audio region.

[0090] In some embodiments, following the example above, assuming the second audio information includes 10 seconds of audio, the audio from the 5th to the 8th second of the 10 seconds of audio is determined as the first recognition result of the target audio region.

[0091] The process of determining the first recognition result described above utilizes visually determined "start and end timestamps" to truncate the audio. This precisely removes all environmental noise, silent segments, and other interference before and after the target speaker begins speaking, achieving accurate alignment and background purity. This ensures that the audio segment fed into the speech recognition engine corresponds exactly to the phonation period, improving input quality. It also enhances the signal-to-noise ratio of speech recognition. By directly replacing the computationally intensive speech activity detection algorithm with visual decision-making, the audio endpoint detection is eliminated, saving computational resources on the automotive chip and reducing truncation errors caused by noise-induced misjudgment of the speech activity detection threshold. The "first video frame" and "last video frame" provide precise time anchors, resolving boundary ambiguity issues caused by reverberation or delay in the audio stream. Clear and unambiguous time boundaries ensure that the final first recognition result corresponds strictly one-to-one with the visual event in terms of time attribution, providing an irrefutable chain of temporal evidence for accurately feeding the recognized text back to the corresponding voice region (e.g., the driver's area).

[0092] In some embodiments, the target object includes a first object and a second object. Based on this, Figure 7 This is another flowchart illustrating the determination of the first identification result provided in the embodiments of this application. Please refer to it. Figure 7 The above step S223 can be achieved through the following steps S231 to S233, which will be explained in detail below.

[0093] Step S231: Based on the motion timestamp of the first object, the second audio information is truncated to obtain the audio information of the first object.

[0094] In some embodiments, following the example above, assuming that the second audio information includes 10 seconds of audio and the motion timestamp of the first object is from the 5th to the 8th second, then the audio from the 5th to the 8th second of the 10 seconds of audio is determined as the audio information of the first object.

[0095] Step S232: The second audio information is truncated based on the motion timestamp of the second object to obtain the audio information of the second object.

[0096] In the embodiments of this application, the motion timestamps of the first object and the second object do not overlap.

[0097] In some embodiments, following the example above, assuming that the second audio information includes 10 seconds of audio and the motion timestamp of the second object is from the 1st to the 4th second, then the audio from the 1st to the 4th second of the 10 seconds of audio is determined as the audio information of the second object.

[0098] Step S233: The audio information of the first object and the audio information of the second object are determined as the first recognition result of the target sound region.

[0099] In some embodiments, the audio information of the first object and the audio information of the second object can be determined as the first recognition result of the target audio region. Alternatively, the audio information of the first object, the motion timestamp of the first object, the audio information of the second object, and the motion timestamp of the second object can be determined as the first recognition result of the target audio region.

[0100] Through the process of determining the first recognition result described above, precise separation and conflict-free splicing of "multi-person time-sharing speech" are achieved within the same audio region, ensuring that regardless of how many speakers are in the region, each can output pure and complete human-computer interaction content. When the second audio information only contains a single audio region (such as the entire microphone in the back row) but two passengers speak one after the other, this embodiment of the application uses visual timestamps to segment the audio into independent segments by person, effectively distinguishing multiple speakers in the same region, fundamentally solving the problem of not being able to distinguish the speaker's affiliation when "multiple people share one audio stream". Since the motion timestamps of the two do not overlap, it indicates that their respective speaking periods are naturally staggered on the time axis. The non-overlapping time ensures the feasibility of physical truncation, which provides a solid basis for pure audio segmentation, avoids complex algorithms for handling speech aliasing (two people speaking at the same time), and makes the truncation operation inherently conflict-free. The truncated first object audio and second object audio are output as a whole result according to their time sequence, preserving the complete semantic order. This not only preserves their independent and clear pronunciation segments, but also fully presents the logical order of the dialogue within the audio region, providing structured temporal data for subsequent speech recognition or semantic understanding. It can be extended to "first object, second object... Nth object", exhibiting good generalization ability. Furthermore, the audio and video evidence of each speaker is independent; even if a brief anomaly occurs in the feature extraction of one path, it will not affect the normal truncation processing of other objects. It supports dynamic expansion of speech recognition and improves the robustness of speech recognition.

[0101] In some embodiments, Figure 8This is another flowchart illustrating the speech recognition method provided in this application embodiment. Please refer to it. Figure 8 Before step S202, the following steps S001 and S002 can also be performed, which are explained in detail below.

[0102] Step S001: Obtain the second video information that is in the same time period as the first audio information.

[0103] In this embodiment of the application, each video frame in the second video information includes at least one detection box of an object to be identified.

[0104] In some embodiments, the video information captured by the mobile device can be truncated based on the time period of the first audio information to obtain second video information that is in the same time period as the first audio information. For example, the time period information of the first audio information and the second video information can be 9:00 AM to 10:00 AM of the same day.

[0105] In the implementation process, the second video information can be considered as video information after object detection. Therefore, each video frame in the second video information also includes a detection box for the object to be identified. This detection box is used to locate and track the object to be identified.

[0106] Step S002: Map the second video information to the audio region to obtain the first video information corresponding to the target audio region.

[0107] In this embodiment of the application, each video frame in the first video information includes a detection box for at least one candidate object, and the candidate object is the object to be identified located in the target audio region.

[0108] In some embodiments, firstly, a preset spatial mapping reference between sound regions and pixels is obtained; then, spatial positioning parameters of each detection box in the second video information are extracted; next, spatial inclusion relationship is determined between the spatial positioning parameters and the pixel coordinate range corresponding to each sound region in the image; finally, data filtering and recombination are performed according to the target sound region to obtain the first video information corresponding to the target sound region.

[0109] Through the aforementioned alignment process between audio and video regions, "spatial dimensionality reduction" and "precise isolation" of visual features are achieved at the "audio region" level, ensuring that the input data for subsequent audio-video fusion processing is strictly aligned in physical space. By mapping audio regions, all object detection boxes belonging to other audio regions (such as the passenger side or the left rear seat) in the global video are filtered out. This effectively avoids cross-regional visual interference caused by the passenger's facial movements when the driver is speaking, ensuring that visual features only serve the correct physical region. Compared to performing full object detection and voice analysis on every frame of the global video, the mapped first video only contains candidate objects within the target audio region. This reduces the computational load and memory consumption required for subsequent "object-by-object voice recognition," reducing computational resources and providing the possibility of real-time processing for low-performance automotive chips. It also ensures that the "second audio information" (focused on the audio region by microphone array beamforming) and the "first video information" (mapped from the camera image) describe the same three-dimensional region in physical space. This rigid alignment of "seeing only what you hear" eliminates the potential for spatial misalignment caused by wide-angle distortion of the camera or microphone pickup delay, laying a solid spatiotemporal foundation for subsequent accurate "audio-visual attribution determination".

[0110] In some embodiments, the first audio information is obtained based on the microphone array of the mobile device. Based on this, Figure 9 This is a flowchart illustrating an information determination method provided in an embodiment of this application. Please refer to it. Figure 9 The above step S201 can be achieved through the following steps S011 to S013, which will be explained in detail below. Step S011: Preprocess the first audio information to obtain processed audio information.

[0111] In some embodiments, preprocessing can be performed using at least one of echo cancellation, noise suppression, dereverberation, automatic gain control, etc., to obtain processed audio information.

[0112] Step S012: Locate the sound source of the processed audio information to obtain the location information of the processed audio information.

[0113] In some embodiments, the continuous digital audio stream is segmented into short, stationary segments with a fixed frame length (e.g., 20 to 30 milliseconds), and a Hamming window is added to reduce spectral leakage. The time difference between sound arrivals at different microphone pairs in the microphone array is then determined. Combined with the physical spacing between the microphone pairs and the speed of sound, the horizontal azimuth and pitch angles of the sound source are inferred using a hyperbolic cross-location method. Furthermore, a delay summation or minimum variance distortion-free response algorithm is used to scan the energy in all directions of space, and the angle where the energy peak is located is determined as the direction of the sound source. Finally, if voices from multiple directions exist simultaneously (e.g., the driver and passenger are speaking at the same time), the system outputs multiple angle values ​​through multi-peak detection.

[0114] Step S013: Map the location information to at least one target audio region; and extract the second audio information corresponding to each target audio region from the first audio information.

[0115] In some embodiments, the azimuth angle output in step S012 is compared with a preset spatial partitioning mapping table. For example, horizontal angles of -45° to -15° are mapped to the "driver's sound zone," +15° to +45° to the "passenger's sound zone," and +120° to +150° to the "rear right-side sound zone." A match is then marked as one of the target sound zones. Next, based on the azimuth angle corresponding to the mapped target sound zone, spatial filtering is performed from the original first audio information (or the signal processed by S011). Spatial filtering may include either fixed beamforming or adaptive beamforming.

[0116] In other embodiments, if there are multiple target audio zones, multiple beamformers will run in parallel and output independent "second audio information" for the driver, passenger, etc., so as to achieve "each listening to their own audio without interfering with each other".

[0117] Through the aforementioned process of determining the target sound zone level second audio information, the physical advantages of the microphone array are utilized to transform the original multi-channel audio into a structured audio stream "divided by spatial region," achieving a precise mapping between the acoustic space and the physical cabin. Preprocessing operations such as noise reduction, dereverberation, and echo cancellation effectively suppress tire noise, wind noise, and multimedia echoes within the vehicle, improving audio signal quality and providing a clean input with a high signal-to-noise ratio for subsequent sound source localization and sound zone extraction, ensuring accurate localization from the source. Sound source localization algorithms (such as time difference of arrival and beamforming) are used to calculate the sound source location, and then the physical angle is mapped to semantic sound zones such as "driver's seat, passenger's seat, rear left / right." This two-step "localization and mapping" strategy allows for flexible decoupling of the microphone array's hardware capabilities from the cabin's functional domain, enabling the same array to adapt to different vehicle seating layouts. The second audio information for each target sound zone is precisely separated from the original first audio information, preparing a private audio channel for subsequent independent "visual-auditory" fusion recognition of each sound zone. This allows voice commands from different positions, such as the driver and passenger, to be processed in parallel without interference, which is the key foundation for achieving "independent multi-zone interaction throughout the vehicle".

[0118] In some embodiments, the first identification result includes target audio information. Based on this, Figure 10 This is another flowchart illustrating the speech recognition method provided in this application embodiment. Please refer to it. Figure 10 After step S202, steps S203 and S204 can also be performed, which will be explained in detail below.

[0119] Step S203: Determine the operation command corresponding to the target audio information.

[0120] In some embodiments, firstly, the target audio information is decoded using speech recognition to convert it into text information. Then, the intent label of the text information is determined using natural language understanding; key entity parameters (i.e., slots) are also extracted from the text information. Finally, the intent label and slot key-value pairs are combined and converted into standardized instructions defined within the mobile device to obtain the operation instructions.

[0121] Step S204: If the operation type of the operation instruction belongs to the preset type set corresponding to the target pitch range, execute the operation instruction.

[0122] In some embodiments, the operation type of the operation instruction is determined by intersecting with a preset set of types for the current target audio region. If the operation type belongs to or is included in the preset set of types, the permission verification is passed, and the operation instruction is issued and executed. If the operation type does not belong to or is not included in the preset set of types, the permission verification is rejected, and the operation instruction is executed.

[0123] By introducing a voice zone-based "permission control" and "function isolation" mechanism, voice interaction is upgraded from "full-domain execution" to "zone-based fine-grained control," improving safety, accuracy, and user experience. By strictly limiting the execution permissions of different seats through preset type sets (e.g., the driver can execute high-priority commands such as navigation and driving mode switching, while the front passenger or rear passengers are prohibited from doing so), the risk of misoperation is avoided, fundamentally preventing children or front passenger passengers from accidentally triggering dangerous commands and ensuring driving safety. It ensures that recognized operation commands are accurately applied to the issuing voice zone (e.g., a rear passenger saying "turn up the temperature" only adjusts the rear air conditioning zone, and a front passenger saying "turn on the seat massage" only activates the front passenger seat), providing a personalized experience that fits the zone's individual needs, avoiding accidental control of the entire vehicle or interference with the driver's seat, and meeting the intuitive expectations of each passenger. Before executing an operation command, it quickly verifies whether the operation type belongs to the "whitelist" of that voice zone, effectively filtering out requests that the location is not authorized to make (e.g., a rear passenger attempting to cancel navigation routes). This reduces the computational burden on subsequent execution modules of the system and avoids interactive confusion caused by frequent rejections or erroneous executions, making human-vehicle dialogue smoother and more compliant with rules, thus improving the user experience.

[0124] The following describes an exemplary application of the embodiments of this application in a real-world application scenario. The example below uses a car as an example of a mobile device.

[0125] Automobiles are typically equipped with an Occupant Monitoring System (OMS) and a Driver Monitoring System (DMS). These two systems continuously acquire video information about the occupants, including facial features, head posture, and mouth area images. This visual information reflects the vocal state of the occupants, providing a new basis for distinguishing between the actual speaker and non-target interference sources within the same vocal range. The actual speaker corresponds to the target object in other embodiments.

[0126] Given that the vehicle-mounted microphone array has already completed sound zone division, the speech detection method in this application embodiment includes how to combine video information provided by OMS and / or DMS to further distinguish multiple potential sound sources within the same sound zone, thereby accurately detecting the real target speaker and outputting the target speaker result or target speech segment for use by the subsequent speech processing module. Here, the real target speaker corresponds to the target object in other embodiments, the target speaker result corresponds to the motion timestamp in other embodiments, and the target speech segment corresponds to the first recognition result in other embodiments.

[0127] This application embodiment achieves target speaker detection based on in-vehicle sound zone pre-separation and visual association with the driver and passenger monitoring system. During implementation, audio information collected by the in-vehicle microphone array and visual information acquired by the in-vehicle driver and passenger monitoring system are simultaneously utilized to perform multimodal association analysis on in-vehicle sound sources, thereby further identifying the true target speaker within the same sound zone environment inside the vehicle.

[0128] In some embodiments, the in-vehicle sound field is first processed and divided into sound zones using a microphone array to obtain candidate audio signals corresponding to each in-vehicle sound zone. Simultaneously, in-vehicle video data is acquired through cameras in the driver and passenger monitoring system, enabling facial recognition and tracking of the driver and passengers, and visual sound zone division based on their positional relationship within the vehicle. Subsequently, the audio sound zones and visual sound zones are consistently categorized to establish a correlation between the in-vehicle driver / passenger and the corresponding sound zone audio. Based on this, a multimodal speaker detection algorithm comprehensively analyzes the visual information of the candidate driver / passenger and the corresponding sound zone audio features to determine the real speaker in the current sound zone, and outputs the detection result of the target speaker and the corresponding speech data. Finally, the detected target speech can be further provided to in-vehicle voice application modules such as speech recognition, intelligent dialogue, or command wake-up. The candidate audio signals correspond to the second audio information in other embodiments, and the in-vehicle video data includes the first video information for each target sound zone.

[0129] Through the above-described speech recognition method, this embodiment of the application can further utilize visual information to distinguish between real speakers and interference sources within the same speech zone, based on the already completed sound zone division of the vehicle microphone array, thereby improving the stability and accuracy of the vehicle voice system in complex in-vehicle environments.

[0130] In some embodiments, Figure 11 This is a schematic diagram of a framework structure of the speech recognition system provided in this application embodiment. Please refer to it. Figure 11 The speech recognition system 1100 includes an audio processing link 1101 and a video processing link 1102. The audio processing link 1101 includes signal processing and microphone array region segmentation, while the video processing link 1102 includes face recognition, face tracking, and visual region segmentation. Based on this, the speech recognition system 1100 also includes consistent region classification 1103 and multimodal speaker detection 1104. Consistent region classification 1103 is used to consistently classify the processing results of the audio processing link 1101 and the video processing link 1102. Multimodal speaker detection 1104 is used to predict the detection results and the corresponding speech segments. Based on this, the speech segments can be used in scenarios such as speech recognition, intelligent dialogue, and command wake-up. They can also be used for voice wake-up, voice enhancement, or other in-vehicle speech processing modules.

[0131] In some embodiments, during the operation of the in-vehicle voice system, multi-channel audio signals from inside the vehicle are acquired via an in-vehicle microphone array, and the acquired audio signals undergo signal processing, such as noise reduction, beamforming, or sound source localization. Based on this, the spatial localization capability of the microphone array is used to divide the in-vehicle sound sources into multiple sound zones, such as the driver's sound zone, the passenger's sound zone, and the rear sound zone, thereby obtaining candidate audio signals corresponding to each in-vehicle sound zone. The audio signals correspond to the first audio information in other embodiments.

[0132] In the implementation process, the division of sound zones can be achieved using different sound source localization or spatial audio processing algorithms.

[0133] In some embodiments, based on the acquired in-vehicle audio signal, in-vehicle video is captured by the camera of the in-vehicle driver and passenger monitoring system, and facial recognition is performed on the video frames to detect the facial positions of the driver and passengers. Subsequently, a facial tracking algorithm is used to continuously track the detected faces, thereby obtaining the facial trajectory information of each driver and passenger in consecutive video frames. Here, "driver and passenger" corresponds to candidate objects in other embodiments.

[0134] In some embodiments, based on the obtained facial trajectory information of the driver and passengers, and combined with the mapping relationship between the in-vehicle camera viewpoint and the spatial position of the seats in the vehicle, the visual and auditory zones of the driver and passengers in the vehicle are divided, thereby determining the visual and auditory zone corresponding to each driver and passenger in the vehicle, such as the driver's area, the front passenger area, or the rear seat area. In the implementation process, driver and passenger detection can be achieved through face detection, human body detection, or other visual recognition algorithms.

[0135] In some embodiments, based on the obtained audio and visual audio information, the audio and visual audio regions are consistently classified, and audio signals from the same spatial region are associated with the corresponding drivers and passengers, thereby establishing a correlation between candidate drivers and passengers and the corresponding audio regions.

[0136] In some embodiments, based on the established association between candidate drivers and audio regions, a neural network-based multimodal speaker detection algorithm comprehensively analyzes the visual information (e.g., mouth movement state) and audio features of the corresponding audio regions of the candidate drivers to determine whether a real speaker exists in the current audio region and identify the specific target driver. Simultaneously, the detection result of the target speaker and the corresponding speech segment are output. In the implementation process, different audio-visual association methods (such as mouth movement detection or audio-visual synchronous analysis) can also be used to determine the target speaker. The speech segment corresponds to the first recognition result in other embodiments.

[0137] In some embodiments, based on the obtained target speaker detection results, the corresponding target voice data is provided to subsequent in-vehicle voice application modules, such as a voice recognition module, an intelligent dialogue system, or a command wake-up module, thereby realizing in-vehicle voice interaction functions.

[0138] The speech recognition method provided in this application, based on the already completed sound zone division by the in-vehicle microphone array, further detects the target speaker for candidate speech within the same sound zone. Simultaneously, using the driver and passenger visual information acquired by the driver and passenger monitoring system, visual sound zone division is performed for the driver and passengers inside the vehicle, and this is correlated and matched with the audio sound zones. Then, based on the sound zone matching, multimodal information is used to determine the actual speaker within the current sound zone. Finally, the target speaker and its corresponding speech data are output and provided to subsequent speech recognition or voice interaction modules. This improves the stability and accuracy of the in-vehicle voice system in complex in-vehicle environments. The speech data corresponds to the first recognition result in other embodiments.

[0139] It should be understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0140] Based on the foregoing embodiments, this application provides a voice recognition device, which includes various modules and units included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor unit (MPU), digital signal processor (DSP) or field-programmable gate array (FPGA), etc.

[0141] Figure 12 This is a schematic diagram of a speech recognition device provided in an embodiment of this application. Please refer to it. Figure 12 In another aspect of the embodiments of this application, a voice recognition device is also provided, which includes: a first determining module 1201 and a second determining module 1202; The first determining module 1201 is used to determine at least one target sound zone located in the internal space of the mobile device based on the first audio information collected by the mobile device, and the second audio information corresponding to each target sound zone in the at least one target sound zone; the internal space of the mobile device is divided into multiple sound zones, and different sound zones correspond to different subspaces in the internal space; The second determining module 1202 is used to determine the first recognition result of the target audio region for each target audio region based on the second audio information and the first video information corresponding to the target audio region.

[0142] In some embodiments, the first video information includes multiple video frames, and each video frame includes at least one candidate object; the second determining module 1202 is further configured to perform voice recognition on each candidate object of the at least one candidate object based on the multiple video frames, and determine the candidate object in the voice state as the target object; and determine the audio information in the second audio information corresponding to the target object as the first recognition result of the target sound region.

[0143] In some embodiments, each video frame further includes a detection box of the at least one candidate object; the second determining module 1202 is further configured to, for each of the at least one candidate object, obtain image information corresponding to the detection box of the candidate object from the plurality of video frames; perform voice recognition based on the image information corresponding to the plurality of detection boxes of the candidate object to obtain a second recognition result of the candidate object; and determine the candidate object as the target object when the second recognition result indicates that the candidate object is in a voice state.

[0144] In some embodiments, the second determining module 1202 is further configured to: extract features from each image information corresponding to the plurality of detection boxes to obtain image features; when the image features indicate that a preset part of the candidate object is in motion, determine the identifier of the video frame corresponding to the image information as a motion identifier; obtain the identifiers of the plurality of video frames; and when the number of consecutive identifiers of the plurality of video frames being the motion identifier is greater than a number threshold, obtain a second recognition result indicating that the candidate object is in a vocalizing state.

[0145] In some embodiments, the second determining module 1202 is further configured to: determine multiple video frames identified by the motion identifier as a target video frame sequence; determine the timestamps corresponding to the first and last video frames in the target video frame sequence as the motion timestamps of the target object; and perform truncation processing on the second audio information based on the motion timestamps to obtain the first recognition result of the target audio region.

[0146] In some embodiments, the target object includes a first object and a second object; the second determining module 1202 is further configured to: truncate the second audio information based on the motion timestamp of the first object to obtain the audio information of the first object; truncate the second audio information based on the motion timestamp of the second object to obtain the audio information of the second object; the motion timestamp of the first object and the motion timestamp of the second object do not overlap; and determine the audio information of the first object and the audio information of the second object as the first recognition result of the target sound region.

[0147] In some embodiments, the speech recognition device further includes a first acquisition module and a mapping module. The first acquisition module is used to acquire second video information that is in the same time period as the first audio information. Each video frame in the second video information includes at least one detection box of an object to be identified. The mapping module is used to perform audio region mapping on the second video information to obtain first video information corresponding to the target audio region. Each video frame in the first video information includes at least one detection box of a candidate object, and the candidate object is an object to be identified that is located in the target audio region.

[0148] In some embodiments, the first audio information is obtained based on the microphone array of the mobile device; the first determining module 1201 is further configured to preprocess the first audio information to obtain processed audio information; perform sound source localization on the processed audio information to obtain the location information of the processed audio information; perform pitch mapping on the location information to obtain the at least one target pitch region; and extract the second audio information corresponding to each target pitch region from the first audio information.

[0149] In some embodiments, the first recognition result includes target audio information; the speech recognition device further includes a third determining module and an execution module, the third determining module being used to determine the operation instruction corresponding to the target audio information; the execution module being used to execute the operation instruction if the operation type of the operation instruction belongs to a preset type set corresponding to the target voice region.

[0150] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0151] It should be noted that, in the embodiments of this application... Figure 12 The module division of the voice recognition device shown is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or be integrated into one unit with two or more units. The integrated units can be implemented in hardware, as software functional units, or a combination of both.

[0152] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a mobile device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0153] Figure 13 This is a schematic diagram of a hardware structure of a mobile device provided in an embodiment of this application. Please refer to it. Figure 13 This application provides a mobile device, which may include, but is not limited to, automobiles, trains, airplanes, ships, construction machinery, sightseeing vehicles, transport vehicles, autonomous shuttle vehicles, mobile cabins, and flying cars. The functions implemented by this method can be achieved by the processor in the mobile device calling program code, which can be stored in a computer storage medium.

[0154] The internal structure diagram of the mobile device can be as follows: Figure 13 As shown, the mobile device includes a processor 1302 and a memory connected via a system bus 1301. The processor 1302 provides computing and control capabilities; it may be, for example, a CPU, or a combination of a CPU and a GPU, etc., without specific limitations. The memory of the mobile device may include a non-volatile storage medium 3031 and internal memory 3032. The non-volatile storage medium 3031 stores an operating system, computer programs, and a database. The internal memory 3032 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium 3031. The database of the mobile device is used to store data. When the computer program is executed by the processor 1302, it implements the aforementioned speech recognition method.

[0155] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0156] This application provides a computer program product containing a computer program or computer-executable instructions, which, when run on a computer, causes the computer to perform the steps in the method provided in the above-described method embodiments.

[0157] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the mobile device to which the present application is applied. A specific mobile device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0158] In one embodiment, the speech recognition device provided in this application can be implemented as a computer program, and the computer program can be implemented in such a way as... Figure 13 The device operates on the mobile device shown. The mobile device's memory can store the various program modules that make up the above-described apparatus. The computer program comprised of the various program modules causes the processor to execute the steps of the methods in the various embodiments of this application described in this specification.

[0159] It should be noted that the descriptions of the computer-readable storage media and mobile devices described above are similar to the descriptions of the method embodiments described above, and have similar beneficial effects. For technical details not disclosed in the storage media, storage media, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0160] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0161] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0162] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0163] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0164] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a mobile device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0165] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0166] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0167] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0168] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech recognition method, characterized in that, Applied to mobile devices, the method includes: Based on the first audio information collected by the mobile device, at least one target sound zone located in the internal space of the mobile device is determined, and second audio information corresponding to each target sound zone in the at least one target sound zone is determined; the internal space of the mobile device is divided into multiple sound zones, and different sound zones correspond to different subspaces in the internal space; For each target audio region, a first recognition result for the target audio region is determined based on the second audio information and the first video information corresponding to the target audio region.

2. The method according to claim 1, characterized in that, The first video information includes multiple video frames, and each video frame includes at least one candidate object; The step of determining the first recognition result of the target audio region based on the second audio information and the first video information corresponding to the target audio region includes: Based on the multiple video frames, voice recognition is performed on each of the at least one candidate object, and the candidate object in the voice state is determined as the target object. The audio information in the second audio information that corresponds to the target object is determined as the first recognition result of the target audio region.

3. The method according to claim 2, characterized in that, Each video frame also includes a detection box for the at least one candidate object; The step of performing voice recognition on each of the at least one candidate object based on the multiple video frames, and determining the candidate object in a voice-emitting state as the target object, includes: For each of the at least one candidate object, image information corresponding to the detection box of the candidate object is obtained from the plurality of video frames; Based on the image information corresponding to multiple detection boxes of the candidate object, voice recognition is performed to obtain a second recognition result of the candidate object; If the second recognition result indicates that the candidate object is in a vocalizing state, the candidate object is identified as the target object.

4. The method according to claim 3, characterized in that, The step of performing speech recognition based on the image information corresponding to multiple detection boxes of the candidate object to obtain a second recognition result of the candidate object includes: For each image information corresponding to the plurality of detection boxes, feature extraction is performed on the image information to obtain image features; When the image features indicate that a preset part of the candidate object is in motion, the identifier of the video frame corresponding to the image information is determined as a motion identifier; thus, the identifiers of the multiple video frames are obtained. If the number of consecutive motion identifiers in the multiple video frames is greater than a threshold, a second recognition result is obtained that indicates that the candidate object is in a vocalizing state.

5. The method according to claim 4, characterized in that, The step of determining the audio information in the second audio information corresponding to the target object as the first recognition result of the target audio region includes: The multiple video frames that are consecutively identified by the motion identifier are determined as the target video frame sequence; The timestamps corresponding to the first and last video frames in the target video frame sequence are determined as the motion timestamps of the target object. The second audio information is truncated based on the motion timestamp to obtain the first recognition result of the target audio region.

6. The method according to claim 5, characterized in that, The target object includes a first object and a second object; The step of truncating the second audio information based on the motion timestamp of the target object to obtain the first recognition result of the target audio region includes: The second audio information is truncated based on the motion timestamp of the first object to obtain the audio information of the first object; The second audio information is truncated based on the motion timestamp of the second object to obtain the audio information of the second object; the motion timestamp of the first object and the motion timestamp of the second object do not overlap. The audio information of the first object and the audio information of the second object are determined as the first recognition result of the target audio region.

7. The method according to any one of claims 1 to 6, characterized in that, Before determining the first recognition result of the target audio region based on the second audio information and the first video information corresponding to the target audio region, the method further includes: Acquire second video information that is in the same time period as the first audio information; each video frame in the second video information includes at least one detection box of the object to be identified; The second video information is mapped to an audio region to obtain the first video information corresponding to the target audio region; each video frame in the first video information includes a detection box of at least one candidate object, and the candidate object is the object to be identified located in the target audio region.

8. The method according to any one of claims 1 to 6, characterized in that, The first audio information is obtained based on the microphone array of the mobile device; The step of determining at least one target sound region located within the internal space of the mobile device based on the first audio information collected by the mobile device, and the second audio information corresponding to each target sound region in the at least one target sound region, includes: The first audio information is preprocessed to obtain processed audio information; The processed audio information is subjected to sound source localization to obtain the location information of the processed audio information; The location information is mapped to a specific area to obtain at least one target area; and second audio information corresponding to each target area is extracted from the first audio information.

9. The method according to any one of claims 1 to 6, characterized in that, The first recognition result includes target audio information; After determining the first recognition result of the target audio region based on the second audio information and the first video information corresponding to the target audio region, the method further includes: Determine the operation command corresponding to the target audio information; If the operation type of the operation instruction belongs to the preset type set corresponding to the target pitch region, the operation instruction is executed.

10. A voice recognition device, characterized in that, The device includes: The first determining module is used to determine at least one target sound zone located in the internal space of the mobile device based on the first audio information collected by the mobile device, and the second audio information corresponding to each target sound zone in the at least one target sound zone; the internal space of the mobile device is divided into multiple sound zones, and different sound zones correspond to different subspaces in the internal space; The second determining module is used to determine a first recognition result of each target sound region based on the second audio information and the first video information corresponding to the target sound region.

11. A mobile device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to implement the speech recognition method as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 9.

Images