Method and apparatus for out-of-vehicle sound reconstruction for in-vehicle zoned perception, and electronic device

Abstract

The application discloses a vehicle exterior sound reconstruction method and device for in-vehicle partition perception and electronic equipment, and mainly has the following design concept: sound pickup and filtering and noise reduction processing are performed by a sound pickup device in a vehicle exterior area, and further, multi-dimensional monitoring of the vehicle exterior environment can be performed by coupling vehicle exterior visual information to guide identification and type judgment of the required external environment sound information for reconstruction; then, different reconstruction modes are distinguished by user instructions, and in an automatic reconstruction scene, the reconstruction sound effects that can be independently perceived are output to each in-vehicle partition, and in a scene with user intervention, the user instructions and user portraits are integrated to realize targeted pushing of the required vehicle exterior environment sound to the target driver and passenger in different in-vehicle partitions, so that the in-vehicle driver and passenger can have a vehicle exterior sound restoration and reproduction effect closer to their daily needs and experience habits, and the high-degree-of-freedom sound perception experience of the driver and passenger and the vehicle safety are improved.

Description

Technical Field

[0001] This invention relates to the field of automotive audio technology, and in particular to a method, apparatus, and electronic device for reconstructing external sounds for in-vehicle zone perception. Background Technology

[0002] With the upgrading of automobile consumption, vehicles equipped with more diverse types of speakers and audio power amplifiers capable of handling complex mixing scenarios are becoming increasingly popular among users, allowing drivers and passengers to enjoy a rich and immersive sound space with equal functionality within the vehicle. However, due to the physical isolation of the vehicle itself and the limitations of various environments, the daily driving convenience and multi-dimensional perception needs of drivers and passengers cannot be met.

[0003] Currently, the main technical solutions for achieving interactive experiences based on in-vehicle and out-of-vehicle spatial perception can be summarized as follows:

[0004] (1) Simultaneously pick up audio from inside and outside the vehicle and compare them to determine the emergency system plan issued inside and outside the vehicle;

[0005] (2) A solution to further eliminate overall noise using external sound pickup data to improve the voice interaction experience;

[0006] (3) External unlocking strategy using external sound pickup for voiceprint matching;

[0007] (4) By recognizing the line-of-sight information of people inside and outside the vehicle, a plan for intercom through the window and emergency call can be realized;

[0008] (5) Use external audio pickup equipment to build a comprehensive and multi-dimensional dashcam system.

[0009] Most of the existing solutions mentioned above use microphones to pick up sound, so that people inside and outside the vehicle can have conversations or verify security information. Their functions are limited. Even when combined with visual acquisition, the purpose is only for simple conversations inside and outside the vehicle. Therefore, they cannot meet the needs of passengers inside the vehicle for sound field reproduction in the area outside the vehicle. Summary of the Invention

[0010] In view of the above, the present invention aims to provide a method, apparatus and electronic device for reconstructing external sound in a vehicle using in-vehicle zone perception, so as to solve the problem of reproducing external environmental sound in the in-vehicle zone.

[0011] The technical solution adopted in this invention is as follows:

[0012] In a first aspect, the present invention provides a method for reconstructing external sound for in-vehicle zone perception, comprising:

[0013] Based on the preset input signal, the external ambient sound reconstruction function is triggered and it is determined whether an intervention command from the user inside the vehicle has been received.

[0014] When no intervention command is received, the preset ambient sound in the area outside the vehicle is collected and identified, and the preset ambient sound is output in each in-vehicle zone based on preset conditions.

[0015] Upon receiving the intervention command, the system picks up and identifies target environmental sounds in the area outside the vehicle based on the intervention command and / or the corresponding user profile obtained.

[0016] Based on the target ambient sound, the reconstructed external ambient sound is output to the vehicle compartment where the user who issued the intervention command is located.

[0017] In at least one possible implementation, the vehicle exterior sound reconstruction method further includes:

[0018] While capturing all ambient sounds outside the vehicle, the system also collects images of the external environment and determines the type of the external environment.

[0019] Based on the external environment type information, the preset environmental sound or the target environmental sound is obtained by filtering from all the collected environmental sounds.

[0020] In at least one possible implementation, obtaining the corresponding user profile includes:

[0021] Upon receiving the intervention command, the vehicle compartment where the current user who issued the intervention command is located is located based on the intervention command, and the identity information of the current user is determined.

[0022] Based on the identity information, construct or retrieve a user profile corresponding to the current user.

[0023] In at least one possible implementation, the step of picking up and identifying target environmental sounds in the area outside the vehicle based on the intervention command and / or the obtained corresponding user profile includes:

[0024] When performing external sound reconstruction, the user intent information represented by the intervention command and / or the user personalization information represented by the user profile are used as weighting factors for sound reconstruction.

[0025] In at least one possible implementation, the preset conditions include deciding whether to output the reconstructed external environmental sound inside the vehicle based on the current vehicle speed, the in-vehicle sound environment, and the external sound pressure level.

[0026] In at least one possible implementation, the vehicle exterior sound reconstruction method further includes:

[0027] If other sound sources are detected in the target vehicle interior partition, the volume of the other sound sources in the target vehicle interior partition is reduced, and the reconstructed external ambient sound is simultaneously isolated and output in the target vehicle interior partition.

[0028] In at least one of the possible implementations, the external sound reconstruction method further includes: dynamically adjusting the output volume of the speakers involved in the in-vehicle partition based on the sound pressure value of the ambient sound in the picked-up external area.

[0029] In a second aspect, the present invention provides an external sound reconstruction device for in-vehicle zone perception, comprising:

[0030] The external ambient sound reconstruction function activation module is used to trigger the external ambient sound reconstruction function based on a preset input signal and to determine whether an intervention command from the user inside the vehicle has been received.

[0031] An automatic reconstruction module is used to collect and identify preset ambient sounds in the area outside the vehicle when no intervention command is received, and to control each in-vehicle zone to output the preset ambient sounds based on preset conditions.

[0032] The target sound acquisition module is used to pick up and identify target environmental sounds in the area outside the vehicle based on the intervention command and / or the corresponding user profile obtained after receiving the intervention command.

[0033] The intervention reconstruction module is used to output the reconstructed external environmental sound to the in-vehicle partition where the user who issued the intervention command is located, based on the target environmental sound.

[0034] In at least one possible implementation, both the automatic reconstruction module and the interventional reconstruction module include an external vision acquisition module, used for:

[0035] While capturing all ambient sounds outside the vehicle, the system also collects images of the external environment and determines the type of the external environment.

[0036] Based on the external environment type information, the preset environmental sound or the target environmental sound is obtained by filtering from all the collected environmental sounds.

[0037] Thirdly, the present invention provides an electronic device, comprising:

[0038] One or more processors, a memory, and one or more computer programs, the memory being a non-volatile storage medium, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the device, cause the device to perform the method as described in the first aspect or any possible implementation thereof.

[0039] The main concept of this invention is to use external sound pickup devices (such as microphone arrays) to pick up and filter noise, and further couple external visual information (such as through a smart cockpit) for multi-dimensional monitoring of the external environment. This guides the identification and type judgment of the external environmental sound information required for reconstruction. In the process of intelligently identifying and filtering environmental sounds from different areas outside the vehicle, the perception dimension and capability are comprehensively improved. Then, user commands distinguish different reconstruction modes. In the automatic reconstruction scenario, the reconstructed sound effects that can be independently perceived are output to each in-vehicle zone. In the scenario with user intervention, the user commands and user profiles are combined to push the external environmental sounds required by the target drivers and passengers in different in-vehicle zones. This brings the external sound reproduction effect that is closer to the daily needs and experience habits of the drivers and passengers, thereby improving the high degree of freedom of sound perception experience and vehicle safety. Attached Figure Description

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:

[0041] Figure 1 A flowchart illustrating an embodiment of the vehicle exterior sound reconstruction method for in-vehicle zone perception provided by the present invention;

[0042] Figure 2 This is a schematic diagram of the vehicle interior partitions and exterior areas provided by the present invention;

[0043] Figure 3 This is a schematic diagram of an embodiment of the vehicle exterior sound reconstruction device for in-vehicle zone perception provided by the present invention;

[0044] Figure 4 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0045] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0046] This invention proposes at least one embodiment of an external sound reconstruction method for in-vehicle zone perception, such as... Figure 1 As shown, it can specifically include:

[0047] Step 1: Based on the preset input signal, trigger the external ambient sound reconstruction function to start, and determine whether the intervention command of the user in the vehicle has been received;

[0048] For example, in practice, a master switch (hardware and software) can be set to enable or disable the entire external ambient sound reconstruction function (which can also be understood as the reproduction of external sound scenes). Typically, this master switch can be set to the off state by default. When the user needs or anticipates needing to perceive external ambient sounds in real time, they can activate the external ambient sound reconstruction function by operating this master switch. Of course, in some preferred embodiments, after the external ambient sound reconstruction function is activated, existing devices such as the vehicle's infotainment system can actively announce selectable control modes, allowing the user to actively decide whether to intervene, and correspondingly keeping the pickup status of several microphones inside and outside the vehicle active.

[0049] The intervention command mentioned here indicates that the user participates in the reconstruction of the external sound. In actual operation, it can specifically refer to listening to the control intentions of one or more users in the car. For example, after the external ambient sound reconstruction function is turned on by the main switch, it monitors whether a voice intervention command is issued by the driver or passenger in a certain car zone within a certain predetermined time, or the vehicle system actively asks the users in the car zone whether they need to push external ambient sound.

[0050] Step 2: When no intervention command is received, collect and identify preset ambient sounds in the area outside the vehicle, and control each in-vehicle zone to output the preset ambient sounds based on preset conditions;

[0051] The interior zones mentioned here can be pre-defined. Taking a small car as an example, they can usually be divided into four interior zones: driver's area S1, passenger's area S2, rear left area S3, and rear right area S4. Each interior zone includes several speakers, such as headrest speakers, low, mid, and high speaker groups, etc. Of course, it is understandable that the common areas in the car can also be equipped with speakers such as a center midrange speaker at the front of the car and a rear subwoofer at the rear of the car. The audio output strategy of these speakers in the common areas can be configured into the aforementioned four specific interior zones as needed.

[0052] Regarding the aforementioned external areas, they can also be divided based on the vehicle model. Taking a small car as an example, they can usually be divided into the front area (F), the rear area (R), the left front area (left-hand drive vehicle, i.e., the outside of the driver's door) (FL), the right front area (left-hand drive vehicle, i.e., the outside of the passenger door) (FR), the left rear area (RL, the outside of the left rear door), and the right rear area (RR, the outside of the right rear door). In specific implementation, microphone arrays, surround-view cameras, etc. can be deployed in each external area.

[0053] Therefore, it can also be mentioned here that the embodiments of the method of the present invention can be composed of hardware and software such as an in-vehicle audio and video control system, an in-vehicle power amplifier system, an in-vehicle system, an in-vehicle and external sound pickup system, an external vision system, and an in-vehicle speaker system, such as, but not limited to, an in-vehicle intelligent cockpit system.

[0054] Specifically, the vehicle audio control system, vehicle amplifier system, and vehicle infotainment system can use the original vehicle configuration. For the hardware solutions of the vehicle's interior and exterior sound pickup system, exterior vision system, and interior speaker system, the following are some preferred examples:

[0055] The preferred in-vehicle speaker system includes headrest speakers, a combination of mid-high and low-frequency speakers in all four doors, a center speaker, and a subwoofer. The installation positions of all speakers can follow standard in-vehicle layouts. In practical application, this example can be adjusted as needed, but it is recommended to include headrest speakers whenever possible; otherwise, the zonal voice interaction feature may not be effectively implemented. The in-vehicle microphone array is arranged to meet the requirements of four-zone voice interaction, such as... Figure 2 Schematic illustration: microphones can be arranged in four areas inside the vehicle (or adjusted as needed); four-zone sound pickup is not the focus of this invention. For the configuration of the external microphone array, preferably, six preset pickup areas should have an array layout (the number of microphones in the array can be considered according to the actual vehicle configuration; understandably, each external pickup area should have at least one microphone. Of course, to ensure better sound pickup and noise reduction, it is preferable to install at least two microphone units in each external pickup area, with a spacing of approximately 30cm between them). As for the vision system, it can also be configured in conjunction with the external sound pickup device as shown in the figure. Of course, in other embodiments, a camera device capable of capturing omnidirectional images of the external environment, such as four fisheye cameras, can also be uniformly configured.

[0056] Regarding the six external areas mentioned above, in conjunction with the appendix Figure 2 This can be illustrated as follows:

[0057] F represents the area in front of the vehicle: for reference, the microphone is mounted near the front camera.

[0058] R (Rear area): For reference, the microphone is installed near the rear camera.

[0059] FL (Front Left Area): For reference, the microphone is mounted near the front left camera.

[0060] FR (Right Front Area): For reference, the microphone is mounted near the right front side camera.

[0061] RL (Rear Left Area): For reference, the microphone is mounted on the left rear door handle or near the left rear side camera.

[0062] RR (Right Rear Area): For reference, the microphone is installed on the right rear door handle or near the right rear side camera.

[0063] Furthermore, in some preferred embodiments, an in-vehicle vision acquisition device is also deployed inside the vehicle, which can capture images of each in-vehicle zone. Of course, it is not limited to setting up independent cameras in each in-vehicle zone. Alternatively, it is possible to acquire global in-vehicle images and then use mature technologies such as person detection and facial recognition to match the users in each in-vehicle zone.

[0064] Furthermore, the preset ambient sounds refer to several target sounds used for reconstructing the external environment sounds of the vehicle, such as, but not limited to, urgent or continuous honking, cries for help or weeping, animal calls, waves crashing, leaves rustling, and other external sounds that have a direct or indirect impact on vehicle movement or occupants. In particular, outputting the preset ambient sounds in this step is equivalent to outputting the reconstructed external sounds. The reconstruction here mainly relies on the selection, noise reduction, mixing, and other characterization processing measures of the directly picked-up external area sounds, and does not involve more complex processing procedures (the subsequent reconstruction process based on user commands and profiles is relatively more complex). The identification of these target sounds can employ mature acoustic processing techniques in the field, such as, but not limited to, extracting acoustic features from the acquired external sounds on demand or according to a predetermined knowledge base through model algorithms and other strategies, and using the information conveyed by the acoustic features to determine the predetermined external sounds for sound reconstruction from the many types of sounds picked up. This invention will not elaborate on this aspect.

[0065] This step can be understood as the intelligent reconstruction of external sounds without user intervention, and the output of these sounds to the vehicle according to designated zones. Two points can be noted here:

[0066] Firstly, the aforementioned "based on preset conditions" can at least refer to deciding whether to output external sounds based on the current vehicle speed, the in-vehicle sound environment, and the external sound pressure level. That is, when the vehicle as a whole or a certain in-vehicle zone / external area meets the aforementioned predetermined conditions, the external sounds can be intelligently controlled in each zone and the common speaker mixing output to reconstruct the external sound field, as described above.

[0067] For example, the vehicle's infotainment system can periodically provide real-time vehicle speed feedback, and the microphone arrays in each external area can assess the external sound pressure level in real time. It can also determine the audio playback status of in-vehicle multimedia through in-vehicle sound pickup or the infotainment system and other in-vehicle terminals. The aforementioned conditions, in this specific implementation, can be as follows: vehicle speed ≤ 20 km / h (indicating parking or low speed), no multimedia being played in the vehicle (to avoid interference), and external sound pressure level ≤ 55 dB (mainly to avoid high noise transmission). Only when these conditions are met will the speakers in different in-vehicle zones output the corresponding external sounds picked up by the microphones in the external areas. Based on this concept, if the above requirements are not met—that is, if the current vehicle speed, in-vehicle sound environment, and external sound pressure level determine that the preset conditions are not met, and a predetermined timeout period (e.g., 5 seconds) is exceeded—the external ambient sound reconstruction function will be automatically turned off. This preferred embodiment can achieve a dynamic intelligent on / off function without requiring manual user intervention.

[0068] Secondly, under the aforementioned premise of intelligent reconstruction, the output of the in-vehicle speaker system can be isolated and output to the corresponding in-vehicle zones. That is, after collecting the preset ambient sound for external sound field reproduction, the audio output in the in-vehicle system should have distinct characteristics, i.e., referencing... Figure 2 For example, sounds acquired in different areas outside the vehicle can be mixed by the vehicle's amplifier and then output independently in different zones within the vehicle:

[0069] Sound pickup in the area outside the vehicle In-car zone mixing output F All speakers in S1 and S2, plus the center speaker FL S1 speaker assembly + headrest speakers FR S2 speaker assembly + headrest speakers R All speakers in S3 and S4 + rear speakers RL S3 speaker array + headrest speakers RR S4 speaker array + headrest speakers

[0070] In this way, occupants in different zones within the vehicle can independently perceive the sounds of their corresponding external areas from their seats (and with the help of mature noise reduction technology, the independent perception effect can be further enhanced). This is similar to the immersive sound effect in a movie theater, where external sound sources in different locations are reproduced inside the vehicle, reflecting the differences in sound effects due to their location. Of course, in other embodiments, the sound energy output from other zones can be adjusted so that occupants, while primarily perceiving the output audio of their own zone, can also hear the output audio of other zones (at relatively lower volumes). This intelligently simulates a more realistic external environmental sound scene inside the vehicle; in other words, global mixing is performed while independently reproducing the sound in each zone. This is, of course, an optional processing step.

[0071] Following the previous text, in step 3, after receiving the intervention command, the target environmental sound in the area outside the vehicle is picked up and identified according to the intervention command and / or the corresponding user profile obtained.

[0072] Step 4: Based on the target ambient sound, output the reconstructed external ambient sound to the in-vehicle partition where the user who issued the intervention command is located.

[0073] There are many mature technologies involved in generating user profile information, such as information recorded during user registration and login, usage history of the vehicle's infotainment system, and mining of personal interests. In actual operation, regardless of whether the user issues the intervention command via voice, touch response to the vehicle's infotainment system, or other interaction methods, the system can obtain the vehicle's current in-vehicle zone (based on acoustic positioning or query response, etc.) and the user's identity and information (through voice recognition or by calling the in-vehicle vision system to collect the image of the user in the corresponding locked in-vehicle zone). Thus, personalized information of the current user can be obtained from pre-stored user data to form their personal "user profile". Of course, it is understood that the user profile can be pre-built and stored for direct retrieval. In some preferred embodiments, the stored user profile can be updated synchronously based on the evolution of big data (such as by using machine learning strategies such as neural networks).

[0074] The concept mentioned in the above two steps involves considering two dimensions when picking up and reconstructing sounds from outside the vehicle (target environmental sounds): the intervention command and the user profile. Specifically, the intervention command provides the sound intent required by the current user, such as a specific sound type and / or a specific area outside the vehicle. For example, a command from a user sitting in the left rear compartment S3 indicates that their intent is to perceive human and animal sounds in the right-side area outside the vehicle (FR+RR). In this case, the information conveyed by the intervention command is used as a higher weight factor in the mixing output during sound reconstruction. The user profile provides auxiliary reconstruction guidance suitable for the characteristics of the current user. For example, a user's interest is to suppress human and animal sounds to a certain extent and highlight natural environmental sounds, or a user's usage habit is to amplify the output volume by a certain factor. That is, it also provides reference weights in the process of external sound reconstruction. In an embodiment that combines the two, based solely on the area outside the vehicle specified by the user in the command and combined with the user profile, only clear natural environmental sounds of that specified area outside the vehicle can be output to the current user to create a unique external sound scene. As described above, the environmental sounds that can be displayed during the reconstruction of external sounds are targeted.

[0075] The following three points can also be added:

[0076] First, similar to the aforementioned intelligent recognition and output of the preset environmental sound, when a user intervenes, the target environmental sound picked up and recognized can also be a specific type of sound filtered out by a pre-built algorithm. In some more preferred embodiments of the present invention, in order to improve the accuracy of external sound reconstruction, especially to accurately reconstruct the external sound scene associated with the user when the user intervenes, visual information of the external environment is integrated when picking up and recognizing external sounds. That is, the external environment information transmitted by the external visual images is combined during the process of reconstructing external sounds inside the vehicle. Specifically, while picking up all environmental sounds in the area outside the vehicle, external environment images (which may, but are not limited to, correspond to the area outside the vehicle where the sound is picked up) are acquired and the type information of the external environment is determined. Based on the type information of the external environment, the preset environmental sound or the target environmental sound is filtered from all the picked-up environmental sounds. Thus, it can be ensured that the external sound to be output reconstructed using the preset environmental sound or the target environmental sound has a higher degree of matching and accuracy with the real external environment.

[0077] It is also worth mentioning that external visual information can be combined with user profiles to identify specific sound types that better match the user's personalized needs within the target vehicle compartment (the compartment where the user who gave the intervention command is located in the vehicle) from all collected external sounds.

[0078] According to the above embodiments, the present invention also proposes that when collecting visual information outside the vehicle, several environmental image data (static or dynamic) can be acquired simultaneously, and when outputting reconstructed sound, the environmental image data can be output simultaneously through the vehicle display terminal to achieve a better experience of external environment reproduction. Of course, it is understandable that the role of the image data is still to assist the user in perceiving the reconstruction of the sound of the external environment.

[0079] Secondly, we can also refer to the above-mentioned approach and consider whether certain preset conditions are met when outputting the reconstructed audio in user-intervention scenarios.

[0080] Third, in some other preferred embodiments, in order to ensure that the user's intention to actively intervene is achieved, the present invention proposes the following preferred concept:

[0081] Step 100: If other sound sources are detected in the target vehicle interior zone (here, "other" refers to the external ambient sound relative to the reconstructed output, such as the speaker in zone S1 playing multimedia sound sources);

[0082] Step 200: Reduce the volume of other audio sources in the target vehicle compartment only (while maintaining the media volume of other vehicle compartments);

[0083] Step 300: Synchronously output the reconstructed external ambient sound in the target vehicle in a zoned isolation, for example, achieving an isolation of more than 10dB in the target zone sound field.

[0084] Finally, based on the aforementioned embodiments, it can be further explained that in actual operation, microphones inside and outside the vehicle can be used to obtain the corresponding noise level and adaptively adjust the audio output effect.

[0085] Specifically, the vehicle amplifier can dynamically adjust the output volume of the in-vehicle zone speakers according to the sound pressure level of the ambient sound outside the vehicle, so that the user can obtain a stable sound volume (e.g., around 50dB, this range can be adjusted as needed). In other embodiments, if the sound pressure level in any in-vehicle zone indicates that the ambient sound output by that zone exceeds a predetermined volume threshold (e.g., the reconstructed sound output to zone S1, after being picked up by the microphone in S1, indicates that it is continuously greater than the set upper limit threshold of 65dB or less than the lower limit threshold of 35dB within 5 seconds), then the output volume of that in-vehicle zone is corrected and a warning is issued; if it cannot be recovered (of course, a timer can be added, for example, if it cannot be corrected to the threshold range within 3 seconds), then the ambient sound reconstruction function is forcibly turned off (and a fault code can be recorded) to ensure that the user can independently perceive a safe and stable sound field reproduction effect.

[0086] In summary, the main concept of this invention lies in using external sound pickup devices (such as microphone arrays) for sound pickup and filtering noise reduction, and further coupling external visual information (such as through a smart cockpit) for multi-dimensional monitoring of the external environment. This guides the identification and type judgment of the external environmental sound information required for reconstruction. During the process of intelligently identifying and filtering environmental sounds from different areas outside the vehicle, the perception dimension and capability are comprehensively improved. Then, user commands distinguish different reconstruction modes. In automatic reconstruction scenarios, independently perceptible reconstruction sound effects are output to each in-vehicle zone. In scenarios with user intervention, user commands and user profiles are integrated to push the required external environmental sounds to target drivers and passengers in different in-vehicle zones. This provides drivers and passengers with an external sound reproduction effect that is closer to their daily needs and experience habits, thereby improving the high degree of freedom of sound perception experience and vehicle safety.

[0087] Corresponding to the above embodiments and preferred solutions, the present invention also provides an embodiment of an external sound reconstruction device for in-vehicle zone perception, such as... Figure 3 As shown, it may specifically include the following components:

[0088] The vehicle exterior sound reconstruction function activation module 1 is used to trigger the activation of the vehicle exterior sound reconstruction function based on a preset input signal, and to determine whether an intervention command from the user inside the vehicle has been received.

[0089] Automatic reconstruction module 2 is used to collect and identify preset ambient sounds in the area outside the vehicle when no intervention command is received, and control each in-vehicle zone to output the preset ambient sounds based on preset conditions;

[0090] The target sound acquisition module 31 is used to pick up and identify target environmental sounds in the area outside the vehicle based on the intervention command and / or the corresponding user profile obtained after receiving the intervention command.

[0091] The intervention reconstruction module 32 is used to output the reconstructed external environmental sound to the in-vehicle partition where the user who issued the intervention command is located, based on the target environmental sound.

[0092] In at least one possible implementation, both the automatic reconstruction module and the interventional reconstruction module include an external vision acquisition module 4, used for:

[0093] While capturing all ambient sounds outside the vehicle, the system also collects images of the external environment and determines the type of the external environment.

[0094] Based on the external environment type information, the preset environmental sound or the target environmental sound is obtained by filtering from all the collected environmental sounds.

[0095] The above should be understood Figure 3 The division of components in the vehicle exterior sound reconstruction device for in-vehicle zone perception shown is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These components can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some components can be implemented in software via processing element calls, while others are implemented in hardware. For example, a particular module can be a separate processing element or integrated into a chip in an electronic device. The implementation of other components is similar. Furthermore, these components can be fully or partially integrated together or implemented independently. During implementation, each step of the above method or each of the above components can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0096] For example, these components can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, these components can be integrated together to form a System-On-a-Chip (SOC).

[0097] Based on the above embodiments and preferred solutions, those skilled in the art will understand that, in actual operation, the technical concept involved in this invention can be applied to various implementation methods. The following embodiments are used as illustrative examples:

[0098] (1) An electronic device. The device may specifically include: one or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the steps / functions of the foregoing embodiments or equivalent embodiments.

[0099] Specifically, the electronic device can be a computer-related electronic device, such as, but not limited to, various interactive terminals and electronic products, mobile terminals, etc.

[0100] Figure 4 The schematic diagram illustrates an embodiment of the electronic device provided by the present invention. Specifically, the electronic device 900 includes a processor 910 and a memory 930. The processor 910 and the memory 930 can communicate with each other via an internal connection to transmit control and / or data signals. The memory 930 stores computer programs, and the processor 910 retrieves and runs the computer programs from the memory 930. The processor 910 and the memory 930 can be combined into a single processing device, but more commonly they are independent components. The processor 910 executes the program code stored in the memory 930 to achieve the aforementioned functions. In specific implementations, the memory 930 can be integrated into the processor 910, or it can be independent of the processor 910.

[0101] In addition, to further enhance the functionality of the electronic device 900, the device 900 may also include one or more of the following: an input unit 960, a display unit 970, an audio circuit 980, a camera 990, and a sensor 901. The audio circuit may also include a speaker 982, a microphone 984, etc. The display unit 970 may include a display screen.

[0102] Furthermore, the aforementioned device 900 may also include a power supply 950 for providing electrical energy to various devices or circuits in the device 900.

[0103] It should be understood that the operation and / or function of the various components in the device 900 can be specifically referred to in the foregoing descriptions of the methods, systems, and other embodiments. To avoid repetition, detailed descriptions are appropriately omitted here.

[0104] It should be understood that Figure 4 The processor 910 in the illustrated electronic device 900 can be a system-on-a-chip (SoC). The processor 910 may include a central processing unit (CPU) and may further include other types of processors, such as a graphics processing unit (GPU), which will be described in detail below.

[0105] In summary, the various processors or processing units inside the processor 910 can work together to implement the previous method flow, and the corresponding software programs of each processor or processing unit can be stored in the memory 930.

[0106] (2) A computer data storage medium storing a computer program or the aforementioned device, which, when executed, causes a computer to perform the steps / functions of the foregoing embodiments or equivalent embodiments.

[0107] In several embodiments provided by this invention, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer data storage medium. Based on this understanding, certain technical solutions of this invention, or the parts that contribute to the prior art, or parts of such technical solutions, can be embodied in the form of software products as described below.

[0108] It should be noted in particular that the storage medium may refer to a server or a similar computer device, specifically, that is, the aforementioned computer program or the aforementioned device is stored in the storage device of the server or similar computer device.

[0109] (3) A computer program product (which may include the above-mentioned device), which, when running on a terminal device, causes the terminal device to execute the vehicle exterior sound reconstruction method for in-vehicle zone perception according to the foregoing embodiments or equivalent embodiments.

[0110] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above implementation methods can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the above-mentioned computer program products may include, but are not limited to, APPs.

[0111] Continuing from the previous text, the aforementioned device / terminal can be a computer device, and the hardware structure of this computer device can specifically include: at least one processor, at least one communication interface, at least one memory, and at least one communication bus; the processor, communication interface, and memory can all communicate with each other through the communication bus. The processor may be a central processing unit (CPU), DSP, microcontroller, or digital signal processor, and may also include a GPU, an embedded neural network processing unit (NPU), and an image signal processor (ISP). The processor may also include a specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. Furthermore, the processor may have the function of operating one or more software programs, which can be stored in a storage medium such as memory. The aforementioned memory / storage medium may include: non-volatile memory, such as a non-removable disk, USB flash drive, portable hard drive, optical disc, etc., as well as read-only memory (ROM), random access memory (RAM), etc.

[0112] In this embodiment of the invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0113] Those skilled in the art will recognize that the modules, units, and method steps described in the embodiments disclosed in this specification can be implemented using electronic hardware, computer software, and a combination of electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0114] Furthermore, the modules and units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed in multiple places, such as nodes in a system network. Specifically, some or all of the modules and units can be selected to achieve the purpose of the above-described embodiments, depending on actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0115] The above description of the structure, features, and effects of the present invention is based on the embodiments shown in the figures. However, the above are only preferred embodiments of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched by those skilled in the art to form a variety of equivalent solutions without departing from or changing the design concept and technical effects of the present invention. Therefore, the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A method for reconstructing external sound for in-vehicle zone perception, characterized in that, include: Based on the preset input signal, the external ambient sound reconstruction function is triggered and it is determined whether an intervention command from the user inside the vehicle has been received. When no intervention command is received, the preset ambient sound in the area outside the vehicle is collected and identified, and the preset ambient sound is output in each in-vehicle zone based on preset conditions. Upon receiving the intervention command, based on the intervention command and the obtained user profile of the current user who issued the intervention command, the target environmental sound in the area outside the vehicle is picked up and identified; Based on the target ambient sound, the reconstructed external ambient sound is output to the vehicle compartment where the user who issued the intervention command is located.

2. The method for reconstructing external sound for in-vehicle zone perception according to claim 1, characterized in that, The method for reconstructing external vehicle sounds also includes: While capturing all ambient sounds outside the vehicle, the system also collects images of the external environment and determines the type of the external environment. Based on the external environment type information, the preset environmental sound or the target environmental sound is obtained by filtering from all the collected environmental sounds.

3. The method for reconstructing external sound for in-vehicle zone perception according to claim 1, characterized in that, The obtained user profile includes: Upon receiving the intervention command, the vehicle compartment where the current user who issued the intervention command is located is located based on the intervention command, and the identity information of the current user is determined. Based on the identity information, construct or retrieve a user profile corresponding to the current user.

4. The method for reconstructing external sound for in-vehicle zone perception according to claim 1, characterized in that, The step of picking up and identifying target environmental sounds in the area outside the vehicle based on the intervention command and the obtained user profile of the current user who issued the intervention command includes: When performing external sound reconstruction, the user intent information represented by the intervention command and the user personalization information represented by the user profile are used as weighting factors for sound reconstruction.

5. The method for reconstructing external sound for in-vehicle zone perception according to any one of claims 1 to 4, characterized in that, The preset conditions include deciding whether to output the reconstructed external environmental sound inside the vehicle based on the current vehicle speed, the in-vehicle sound environment, and the external sound pressure level.

6. The method for reconstructing external sound for in-vehicle zone perception according to any one of claims 1 to 4, characterized in that, The method for reconstructing external vehicle sounds also includes: If other sound sources are detected in the target vehicle interior partition, the volume of the other sound sources in the target vehicle interior partition is reduced, and the reconstructed external ambient sound is simultaneously isolated and output in the target vehicle interior partition.

7. The method for reconstructing external sound for in-vehicle zone perception according to any one of claims 1 to 4, characterized in that, The method for reconstructing external sound also includes: dynamically adjusting the output volume of the speakers involved in the vehicle's internal zones based on the sound pressure level of the ambient sound in the external area.

8. A device for reconstructing external sound for in-vehicle zone perception, characterized in that, include: The external ambient sound reconstruction function activation module is used to trigger the external ambient sound reconstruction function based on a preset input signal and to determine whether an intervention command from the user inside the vehicle has been received. An automatic reconstruction module is used to collect and identify preset ambient sounds in the area outside the vehicle when no intervention command is received, and to control each in-vehicle zone to output the preset ambient sounds based on preset conditions. The target sound acquisition module is used to pick up and identify target environmental sounds in the area outside the vehicle after receiving the intervention command, based on the intervention command and the user profile of the current user who issued the intervention command. The intervention reconstruction module is used to output the reconstructed external environmental sound to the in-vehicle partition where the user who issued the intervention command is located, based on the target environmental sound.

9. The external sound reconstruction device for in-vehicle zone perception according to claim 8, characterized in that, Both the automatic reconstruction module and the interventional reconstruction module include an external vision acquisition module, used for: While capturing all ambient sounds outside the vehicle, the system also collects images of the external environment and determines the type of the external environment. Based on the external environment type information, the preset environmental sound or the target environmental sound is obtained by filtering from all the collected environmental sounds.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method for reconstructing external sound for in-vehicle zone perception as described in any one of claims 1 to 7.

11. A computer data storage medium, characterized in that, The computer data storage medium stores a computer program, which, when run on the computer, causes the computer to execute the method for reconstructing external sounds for in-vehicle zone perception as described in any one of claims 1 to 7.

Images