Audio playing method and system based on camera, electronic equipment and medium

Abstract

The invention provides an audio playing method and system based on a camera, electronic equipment and a medium, and belongs to the technical field of intelligent cabin audio interaction, and the method comprises the steps: obtaining a target user ear multi-view image collected by the camera, the method comprises the following steps: detecting auricle feature points of a target user in real time by adopting a lightweight CNN model based on an ear multi-view image of the target user, and solving real-time 6DoF pose parameter data of the target user through a PnP algorithm in combination with a preset 3D mesh model of the target user; and obtaining PCM audio data, rendering the PCM audio data through a spatialization engine in combination with the real-time 6DoF pose parameter data of the target user, and outputting the PCM audio data with a spatial orientation sense for broadcasting. According to the invention, through fusion of visual positioning of the camera and the spatialization sound field engine, immersive and high-quality spatial audio experience is provided for passengers in a vehicle.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit audio interaction technology, and in particular to a camera-based audio playback method and system, electronic device, and medium. Background Technology

[0002] With the rapid development of smart cockpits and in-vehicle entertainment systems, users' demand for immersive spatial audio experiences is growing. Traditional in-vehicle audio systems typically use a fixed speaker layout, and their sound field performance is limited by the physical speaker positions. They cannot dynamically adjust according to the actual position and head orientation of passengers, thus failing to achieve an immersive, high-quality spatial audio experience. Summary of the Invention

[0003] This invention aims to solve at least one of the aforementioned problems in the prior art and proposes a camera-based in-vehicle spatial audio playback method. This method is based on the hardware provided by the cameras of the in-vehicle driver monitoring system (DMS) and occupant monitoring system (OMS). It uses a lightweight convolutional neural network (CNN) model to detect the auricular feature points of passengers in real time, calculates the 6DoF pose (position XYZ + rotation RPY), and establishes personalized modeling of head-related transfer function (HRTF) based on the seat position. It achieves the fusion of camera visual positioning and spatial sound field engine, providing in-vehicle passengers with an immersive and high-quality spatial audio experience without increasing hardware costs.

[0004] In a first aspect, embodiments of the present invention provide an audio playback method based on a camera, comprising:

[0005] The system acquires multi-view images of the target user's ear captured by the camera. Based on these multi-view images, a lightweight CNN model is used to detect the auricular feature points of the target user in real time. Combined with a pre-set 3D mesh model of the target user, the PnP algorithm is used to solve for the real-time 6DoF pose parameter data of the target user.

[0006] Acquire PCM audio data, combine it with the target user's real-time 6DoF pose parameter data, render the PCM audio data through a spatialization engine, and output PCM audio data with spatial orientation for playback.

[0007] In a preferred embodiment, it further includes:

[0008] Acquire spectral imaging images around the target user's ear and generate a preset 3D mesh model of the target user.

[0009] In a preferred embodiment, the step of acquiring images captured by spectral imaging around the target user's ear and generating a preset 3D mesh model of the target user includes:

[0010] Acquire images captured by multiple cameras in the vehicle's DMS / OMS system around the target user's ear using spectral imaging, generate a 360° surround image sequence of the target user, and record the target user's 6DoF pose parameter data.

[0011] Obtain the target user's 3D auricle point cloud based on the surrounding image sequence, output the preset target user 3D mesh model based on the 3D coordinates of the anatomical feature points annotated on the 3D auricle point cloud, and encapsulate it into a standardized HRTF database file.

[0012] In a preferred embodiment, the steps of acquiring multi-view images of the target user's ear captured by the camera, detecting auricular feature points of the target user in real time using a lightweight CNN model based on the multi-view images of the target user's ear, and solving the real-time 6DoF pose parameter data of the target user using the PnP algorithm in combination with a preset 3D mesh model of the target user include:

[0013] Acquire multi-view images of the target user's ear captured by the camera, capture auricular feature images based on the multi-view images, and obtain cropped and corrected auricular feature images;

[0014] The auricle feature image is input into a lightweight CNN model to obtain key feature points. Projecting these points onto a plane yields 2D feature point coordinates. The 2D feature point coordinates are then... i =(u1,u2,...,u n ) and the preset target user 3D mesh model 3D feature point coordinates X i =(X1,X2,...,X n Matching is performed to form 2D-3D corresponding point pairs. The PnP geometric algorithm is combined with the camera projection model π to establish an objective function to solve for the rotation matrix R and translation vector t in the camera coordinate system, and output the real-time 6DoF pose parameter data of the target user.

[0015] The objective function is expressed by the following formula: ,

[0016] Among them, X i For 3D model points, u i π represents the 2D detection point, and π represents the camera projection model.

[0017] In a preferred embodiment, the step of acquiring multi-view images of the target user's ear captured by the camera, using a lightweight CNN model to detect auricular feature points in real time based on the multi-view images of the target user's ear, and combining the preset 3D mesh model of the target user with the PnP algorithm to solve the real-time 6DoF pose parameter data of the target user, further includes:

[0018] The Levenberg-Marquardt algorithm is used to optimize the real-time 6DoF pose parameter data of the target user.

[0019] Based on the optimized real-time 6DoF pose parameter data of the target user, a Kalman filter prediction-correction process is used to eliminate instantaneous jitter in single-frame detection, and the optimized real-time 6DoF pose parameter data of the target user is output.

[0020] In a preferred embodiment, the step of optimizing the real-time 6DoF pose parameter data of the target user using the Levenberg-Marquardt algorithm includes:

[0021] The distance constraint term for setting the distance between the ear canal entrance and the center of the head is: , where t ear Let t be the 3D coordinate of the ear canal entrance in the head coordinate system. head 3D coordinates of the head center;

[0022] The angle constraint term for the angle between the auricle normal vector and the head orientation is set as follows:

[0023] ,

[0024] ,

[0025] Where θ tilt Let n be the normal vector of the auricle. near With head orientation vector n head The included angle;

[0026] The formula for the Levenberg-Marquardt method is as follows:

[0027] ,

[0028] The distance constraint penalty term is as follows: The angle constraint penalty term is: λ is the weight controlling physiological constraints, and C(R,t) is the optimized real-time 6DoF pose parameter data of the target user.

[0029] In a preferred embodiment, the step of eliminating instantaneous jitter in single-frame detection using a Kalman filter prediction-correction process based on the optimized real-time 6DoF pose parameter data of the target user, and outputting the optimized real-time 6DoF pose parameter data of the target user, includes:

[0030] Using the optimized real-time 6DoF pose parameter data of the target user as the observation value, and supplemented by the Kalman filter prediction-correction process to eliminate the instantaneous jitter of single-frame detection, the optimized real-time 6DoF pose parameter data of the target user is output.

[0031] The formula used in the Kalman filter prediction-correction process is:

[0032] ,

[0033] Wherein, the 3D position of the head in the camera coordinate system is x, y, z, and the linear velocity of the head is... , , This provides optimized real-time 6DoF pose parameter data for the target user. , , , is the head tilt angle, Z K To optimize the real-time 6DoF pose parameter data for the target user.

[0034] In a second aspect, embodiments of the present invention provide a camera-based audio playback system, the system being configured to implement any of the methods described in the first aspect, the system comprising:

[0035] The image acquisition and processing module is used to acquire multi-view images of the target user's ear captured by the camera. Based on the multi-view images of the target user's ear, a lightweight CNN model is used to detect the auricular feature points of the target user in real time. Combined with the preset 3D mesh model of the target user, the PnP algorithm is used to solve the real-time 6DoF pose parameter data of the target user.

[0036] The spatial audio module is used to acquire PCM audio data, combine it with the target user's real-time 6DoF pose parameter data, render the PCM audio data through a spatialization engine, and output PCM audio data with spatial orientation for playback.

[0037] Thirdly, embodiments of the present invention provide an electronic device, including:

[0038] One or more processors;

[0039] Memory, used to store one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the first aspect.

[0041] Fourthly, embodiments of the present invention provide a computer-readable medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0042] Beneficial effects of this invention:

[0043] This invention provides in-vehicle passengers with an immersive, high-quality spatial audio experience by integrating camera visual positioning with a spatial sound field engine.

[0044] This invention reuses the vehicle-mounted DMS / OMS camera and vehicle-mounted computing unit, eliminating the need for additional dedicated sensors and reducing the hardware cost of spatial audio technology.

[0045] This invention achieves a 6DoF-based pose dynamic tracking accuracy far exceeding that of traditional IMU (Inertial Measurement Unit) schemes through a visual-auricular biometric fusion algorithm. At the same time, the sound field rendering maintains strict spatiotemporal consistency with head movement, eliminating the problem of inertial measurement lag in traditional schemes.

[0046] This invention introduces a user-based personalized spatial audio subscription service, which generates a unique HRTF profile for users through 3D ear scanning, creating a commercial profit model for spatial audio. At the same time, through the experience of "sound following the head, a thousand people, a thousand sounds", it has become a core selling point for high-end car models. Attached Figure Description

[0047] Figure 1 This is one of the schematic diagrams of an audio playback method based on a camera provided in an embodiment of the present invention.

[0048] Figure 2 This is a second schematic diagram of a camera-based audio playback method provided in an embodiment of the present invention.

[0049] Figure 3 This is a flowchart illustrating an optional specific implementation method of step S0 in an embodiment of the present invention.

[0050] Figure 4 This is a flowchart illustrating an optional specific implementation method of step S1 in an embodiment of the present invention.

[0051] Figure 5 This is a schematic diagram of a camera-based audio playback system provided in an embodiment of the present invention.

[0052] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0054] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0055] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0056] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0057] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0058] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0059] In this invention, some technical terms have the following meanings:

[0060] DMS, Driver Monitor System, in-vehicle driver monitoring system;

[0061] OMS, Occupancy Monitoring System;

[0062] CNN, Convolutional Neural Networks, is a lightweight convolutional neural network.

[0063] 6DoF, Six Degrees of Freedom Head Tracking, tracks all six types of head movements (up / down, left / right, forward / backward, pitch, yaw, and roll).

[0064] 6DoF attitude combines the position and orientation information of an object to fully describe its "position and orientation" in three-dimensional space. Position is represented by (X, Y, Z) coordinates, and orientation is represented by (R, P, Y) corresponding to roll angle (Roll), pitch angle (Pitch), and yaw angle (Yaw).

[0065] PnP, Perspective-n-Point, is a perspective n-point algorithm.

[0066] HRTF, Head Related Transfer Function.

[0067] One related technology, Chinese patent document CN116744216A, discloses a technical solution for "A Virtual Surround Sound Audio System and Design Method for Automobile Space Based on Binaural Effect". The main process of this solution is as follows: by establishing a coordinate system in the automobile space, the orientation information of the target left and right ears and each speaker is determined; the HRTF response of each speaker is determined as a function of the sound signal source frequency; the sum of the left and right ear responses generated by all speakers is calculated by summation; and an optimization algorithm is used to determine the target frequency of the signal source corresponding to each speaker, which is then used to control the operating frequency of each speaker. The main drawbacks of this solution are: 1. This solution is based on fixed left and right ear orientation parameters and does not consider the orientation shift caused by the user's movements during actual travel in a vehicle; 2. This solution does not disclose how to determine the orientation parameters of the user's left and right ears.

[0068] The second related technology is disclosed in Chinese patent document CN104765444A, which presents a technical solution for an "in-vehicle gesture interaction spatial audio system." The main process of this solution is: acquiring sensor data associated with the user's hand, analyzing the sensor data to determine at least one hand position, modifying spatial audio parameters associated with the audio event based on the interaction to generate a modified audio stream, and reproducing the modified audio stream for output to the user. The main drawbacks of this solution are: 1. This solution relies on gestures to determine the user's position, failing to achieve a high-quality spatial audio experience based on HRTF precise positioning; 2. This solution outputs spatial audio through the user's gestures, requiring the user to actively perform gesture operations, thus failing to achieve a dynamic, real-time spatial audio experience within the vehicle.

[0069] In summary, the in-vehicle spatial audio playback technology involved in the relevant technologies still has many problems and cannot achieve an immersive, high-quality spatial audio experience.

[0070] Figure 1 This is one of the flowcharts illustrating a camera-based audio playback method provided in an embodiment of the present invention; as shown below. Figure 1 As shown, the method includes:

[0071] Step S1: Acquire multi-view images of the target user's ear captured by the camera. Based on the multi-view images of the target user's ear, a lightweight CNN model is used to detect the auricular feature points of the target user in real time. Combined with the preset 3D mesh model of the target user, the real-time 6DoF pose parameter data of the target user is solved by the PnP algorithm.

[0072] Step S2: Acquire PCM audio data, combine it with the target user's real-time 6DoF pose parameter data, render the PCM audio data through a spatialization engine, output PCM audio data with spatial orientation for playback, and finally play it out through the vehicle's speakers to output high-precision spatialized audio.

[0073] In some embodiments, such as Figure 2 As shown, it also includes:

[0074] Step S0: Acquire images of the target user's ear using spectral imaging and generate a preset 3D mesh model of the target user.

[0075] In some embodiments, such as Figure 3 As shown, step S0, acquiring an image of the target user's ear using spectral imaging and generating a preset 3D mesh model of the target user, includes:

[0076] Step S01: Acquire images captured by the vehicle's DMS / OMS multi-camera system around the target user's ear using spectral imaging, generate a 360° surround image sequence of the target user, and record the target user's 6DoF pose parameter data.

[0077] Step S02: Obtain the target user's 3D auricle point cloud established based on the surrounding image sequence. Output the preset target user's 3D mesh model based on the 3D coordinates of the anatomical feature points (such as the ear canal entrance, tragus, antihelix, etc.) annotated on the 3D auricle point cloud, and encapsulate it into a standardized HRTF database file.

[0078] By scanning a user's 3D ear model, a configuration file based on the user's unique HRTF database is generated, which solves the spatial positioning error problem caused by physiological differences such as ear shape and head shape in general HRTF.

[0079] To protect user privacy, the HRTF database file is encrypted and stored in the vehicle's TEE (Trusted Execution Environment) to block unauthorized access and is not uploaded to the cloud.

[0080] In some embodiments, such as Figure 4 As shown, step S1, acquiring multi-view images of the target user's ear captured by the camera, using a lightweight CNN model to detect the target user's auricle feature points in real time based on the multi-view images, and combining the preset target user 3D mesh model, using the PnP algorithm to solve the real-time 6DoF pose parameter data of the target user, includes the following steps:

[0081] Step S11: Obtain multi-view images of the target user's ear captured by the camera, capture auricular feature images based on the multi-view images, and obtain cropped and corrected auricular feature images.

[0082] Step S12: Input the auricle feature image into a lightweight CNN model to obtain key feature points. Project the key feature points onto a plane to obtain 2D feature point coordinates. Then, input the 2D feature point coordinates u... i =(u1,u2,...,u n ) and the preset target user 3D mesh model 3D feature point coordinates X i =(X1,X2,...,X n Matching is performed to form 2D-3D corresponding point pairs. The PnP geometric algorithm is combined with the camera projection model π to establish an objective function to solve for the rotation matrix R and translation vector t in the camera coordinate system, and output the real-time 6DoF pose parameter data of the target user.

[0083] The objective function is expressed by the following formula: ,

[0084] Among them, X i For 3D model points, ui For 2D detection points, π is the camera projection model, and the output is 6DoF pose parameter data (translation t, rotation R). The objective function is to minimize the reprojection error.

[0085] In some embodiments, such as Figure 4 As shown, step S1, acquiring multi-view images of the target user's ear captured by the camera, using a lightweight CNN model to detect the target user's auricular feature points in real time based on the multi-view images, and combining the preset target user 3D mesh model, further includes solving the target user's real-time 6DoF pose parameter data using the PnP algorithm:

[0086] Step S13: Optimize the real-time 6DoF pose parameter data of the target user using the Levenberg-Marquardt algorithm;

[0087] Step S14: Based on the optimized real-time 6DoF pose parameter data of the target user, the instantaneous jitter of single-frame detection is eliminated by Kalman filtering prediction-correction process, and the optimized real-time 6DoF pose parameter data of the target user is output.

[0088] In some embodiments, step S13, which involves optimizing the real-time 6DoF pose parameter data of the target user using the Levenberg-Marquardt algorithm, includes:

[0089] The distance constraint term for setting the distance between the ear canal entrance and the center of the head is: , where t ear Let t be the 3D coordinate of the ear canal entrance in the head coordinate system. head 3D coordinates of the head center;

[0090] The angle constraint term for the angle between the auricle normal vector and the head orientation is set as follows:

[0091] ,

[0092] ,

[0093] Where θ tilt Let n be the normal vector of the auricle. near With head orientation vector n head The included angle;

[0094] The formula for the Levenberg-Marquardt method is as follows:

[0095] ,

[0096] The distance constraint penalty term is as follows: The angle constraint penalty term is: λ is the weight controlling physiological constraints, and C(R,t) is the optimized real-time 6DoF pose parameter data of the target user.

[0097] In some embodiments, step S14, which involves eliminating instantaneous jitter in single-frame detection using a Kalman filter prediction-correction process based on the optimized real-time 6DoF pose parameter data of the target user, and outputting the optimized real-time 6DoF pose parameter data of the target user, includes:

[0098] Using the optimized real-time 6DoF pose parameter data of the target user as the observation value, and supplemented by the Kalman filter prediction-correction process to eliminate the instantaneous jitter of single-frame detection, the optimized real-time 6DoF pose parameter data of the target user is output.

[0099] The formula used in the Kalman filter prediction-correction process is:

[0100] ,

[0101] Wherein, the 3D position of the head in the camera coordinate system is x, y, z, and the linear velocity of the head is... , , This provides optimized real-time 6DoF pose parameter data for the target user. , , , is the head tilt angle, Z K To optimize the real-time 6DoF pose parameter data for the target user.

[0102] Based on the same inventive concept, embodiments of the present invention also provide a camera-based audio playback system, the system being configured to implement any of the methods described in the above embodiments, such as... Figure 5 As shown, the system includes:

[0103] The image acquisition and processing module is used to acquire multi-view images of the target user's ear captured by the camera. Based on the multi-view images of the target user's ear, a lightweight CNN model is used to detect the auricular feature points of the target user in real time. Combined with the preset 3D mesh model of the target user, the PnP algorithm is used to solve the real-time 6DoF pose parameter data of the target user.

[0104] The spatial audio module is used to acquire PCM audio data, combine it with the target user's real-time 6DoF pose parameter data, render the PCM audio data through a spatialization engine, and output PCM audio data with spatial orientation for playback.

[0105] The spatial audio module periodically receives 6DoF parameters output from the image acquisition and processing module for use by the spatialized sound field engine, providing real-time spatial positioning data support for sound field modeling. Using multi-track audio sources with standards such as Dolby Atmos or Audio Vivid, it decodes the multi-track audio sources into PCM audio data. Employing real-time acquired 6DoF parameters and calling the user-specific HRTF database, it performs low-latency audio rendering through the dynamic spatialized sound field engine, achieving spatial synchronization of sound location, distance, and head movement. This results in a high-precision PCM audio stream with spatial effects, preserving the three-dimensional spatial information of the sound field. The PCM audio stream is written to a designated PCM device node, where it undergoes multi-channel mixing, equalization adjustment, and dynamic range compression by the digital signal processing unit (DSP). Finally, it is played through the vehicle's multi-zone speaker system for panoramic sound field playback, ensuring a immersive spatial experience within the vehicle's acoustic environment.

[0106] In some embodiments, such as Figure 5 As shown, the system also includes:

[0107] The scanning module is used to acquire images of the target user's ear through spectral imaging and generate a preset 3D mesh model of the target user.

[0108] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 6 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0109] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0110] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0111] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0112] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable medium. This computer-readable medium stores a computer program, wherein, when executed by a processor, the program implements the steps of any of the methods described in the above embodiments. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0113] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0114] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0115] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0116] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0117] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0118] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0119] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0120] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0122] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A camera-based audio playback method, characterized in that, include: The system acquires multi-view images of the target user's ear captured by the camera. Based on these multi-view images, a lightweight CNN model is used to detect the auricular feature points of the target user in real time. Combined with a pre-set 3D mesh model of the target user, the PnP algorithm is used to solve for the real-time 6DoF pose parameter data of the target user. Acquire PCM audio data, combine it with the target user's real-time 6DoF pose parameter data, render the PCM audio data through a spatialization engine, and output PCM audio data with spatial orientation for playback.

2. The method according to claim 1, characterized in that, Also includes: Acquire spectral imaging images around the target user's ear and generate a preset 3D mesh model of the target user.

3. The method according to claim 2, characterized in that, The step of acquiring spectral imaging images around the target user's ear and generating a preset 3D mesh model of the target user includes: Acquire images captured by multiple cameras in the vehicle's DMS / OMS system around the target user's ear using spectral imaging, generate a 360° surround image sequence of the target user, and record the target user's 6DoF pose parameter data. Obtain the target user's 3D auricle point cloud based on the surrounding image sequence, output the preset target user 3D mesh model based on the 3D coordinates of the anatomical feature points annotated on the 3D auricle point cloud, and encapsulate it into a standardized HRTF database file.

4. The method according to claim 1, 2 or 3, characterized in that, The steps of acquiring multi-view images of the target user's ear captured by the camera, using a lightweight CNN model to detect auricular feature points in real time based on the multi-view images, and combining them with a pre-set 3D mesh model of the target user to solve the real-time 6DoF pose parameter data of the target user through the PnP algorithm include: Acquire multi-view images of the target user's ear captured by the camera, capture auricular feature images based on the multi-view images, and obtain cropped and corrected auricular feature images; The auricle feature image is input into a lightweight CNN model to obtain key feature points. Projecting these points onto a plane yields 2D feature point coordinates. The 2D feature point coordinates are then... i =(u1,u2,...,u n ) and the preset target user 3D mesh model 3D feature point coordinates X i =(X1,X2,...,X n Matching is performed to form 2D-3D corresponding point pairs. The PnP geometric algorithm is combined with the camera projection model π to establish an objective function to solve for the rotation matrix R and translation vector t in the camera coordinate system, and output the real-time 6DoF pose parameter data of the target user. The objective function is expressed by the following formula: , Among them, X i For 3D model points, u i π represents the 2D detection point, and π represents the camera projection model.

5. The method according to claim 4, characterized in that, The steps of acquiring multi-view images of the target user's ear captured by the camera, using a lightweight CNN model to detect auricular feature points in real time based on the multi-view images, and combining them with a pre-set 3D mesh model of the target user to solve the real-time 6DoF pose parameter data of the target user using the PnP algorithm, further include: The Levenberg-Marquardt algorithm is used to optimize the real-time 6DoF pose parameter data of the target user. Based on the optimized real-time 6DoF pose parameter data of the target user, a Kalman filter prediction-correction process is used to eliminate instantaneous jitter in single-frame detection, and the optimized real-time 6DoF pose parameter data of the target user is output.

6. The method according to claim 5, characterized in that, The steps for optimizing the real-time 6DoF pose parameter data of the target user using the Levenberg-Marquardt algorithm include: The distance constraint term for setting the distance between the ear canal entrance and the center of the head is: , where t ear Let t be the 3D coordinate of the ear canal entrance in the head coordinate system. head 3D coordinates of the head center; The angle constraint term for the angle between the auricle normal vector and the head orientation is set as follows: ， ， Where θ tilt Let n be the normal vector of the auricle. near With head orientation vector n head The included angle; The formula for the Levenberg-Marquardt method is as follows: ， The distance constraint penalty term is as follows: The angle constraint penalty term is: λ is the weight for controlling physiological constraints, and C(R,t) is the optimized real-time 6DoF pose parameter data of the target user.

7. The method according to claim 5, characterized in that, The step of eliminating instantaneous jitter in single-frame detection using a Kalman filter prediction-correction process based on the optimized real-time 6DoF pose parameter data of the target user, and outputting the optimal real-time 6DoF pose parameter data of the target user, includes: Using the optimized real-time 6DoF pose parameter data of the target user as the observation value, and supplemented by the Kalman filter prediction-correction process to eliminate the instantaneous jitter of single-frame detection, the optimized real-time 6DoF pose parameter data of the target user is output. The formula used in the Kalman filter prediction-correction process is: ， Wherein, the 3D position of the head in the camera coordinate system is x, y, z, and the linear velocity of the head is... , , This provides optimized real-time 6DoF pose parameter data for the target user. , , , is the head tilt angle, Z K To optimize the real-time 6DoF pose parameter data for the target user.

8. An audio playback system based on a camera, characterized in that, The system is configured to implement the method as described in any one of claims 1 to 7, the system comprising: The image acquisition and processing module is used to acquire multi-view images of the target user's ear captured by the camera. Based on the multi-view images of the target user's ear, a lightweight CNN model is used to detect the auricular feature points of the target user in real time. Combined with the preset 3D mesh model of the target user, the PnP algorithm is used to solve the real-time 6DoF pose parameter data of the target user. The spatial audio module is used to acquire PCM audio data, combine it with the target user's real-time 6DoF pose parameter data, render the PCM audio data through a spatialization engine, and output PCM audio data with spatial orientation for playback.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images