HRTF implementation method, apparatus and device, and computer readable storage medium

Abstract

The invention discloses an HRTF implementation method, device and equipment and a computer readable storage medium, relates to the technical field of HRTFs, is applied to an earphone terminal for implementing the HRTF and configuring a camera, and comprises the steps that image information collected by the camera is acquired, and the image information comprises a wearing process image in the wearing process of the earphone terminal; determining a human body region image according to the wearing process image and a preset human body region division basis; and determining a target HRTF parameter according to the human body region image so as to realize HRTF based on the target HRTF parameter. According to the invention, the intelligence of HRTF realization is improved.

Description

Technical Field

[0001] The present application relates to the field of HRTF technology, and in particular to a method, device, equipment and computer-readable storage medium for implementing HRTF. Background Art

[0002] With the development of HRTF (Head-Related Transfer Function), users have also put forward higher requirements for the implementation of HRTF.

[0003] The traditional way to implement HRTF is to take a static photo of the user through the camera and scanning function of an external device, and then implement HRTF based on the static photo image (that is, the user and the external device are both static). This way of implementing HRTF has great defects. It can only be implemented through external devices and for static photos, which makes the HRTF implementation not very intelligent.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a method, device, equipment and computer-readable storage medium for implementing HRTF, aiming to solve the technical problem that the intelligence of HRTF implementation is not high.

[0006] To achieve the above-mentioned purpose, the present application provides a method for implementing HRTF, which is applied to a headphone terminal that implements HRTF and is equipped with a camera, and the method for implementing HRTF includes:

[0007] Acquire image information collected by the camera, wherein the image information includes a wearing process image of the headset terminal during wearing;

[0008] Determine a human body region image according to the wearing process image and a preset human body region division basis;

[0009] Target HRTF parameters are determined according to the human body region image to implement HRTF based on the target HRTF parameters.

[0010] In one embodiment, the headset terminal includes a position sensor, the human body region image includes an ear canal image, the human body region division basis includes a preset wearing completion state, and the step of determining the human body region image according to the wearing process image and the preset human body region division basis includes:

[0011] Determine the target position information corresponding to each frame of the wearing process image in sequence, wherein the target position information includes the position state collected by the position sensor;

[0012] When the position state matches the wearing completion state, a wearing process image corresponding to the target position information is determined as an ear canal image.

[0013] In one embodiment, the human body region image includes a torso image and a head image, the human body region division basis includes a preset torso training feature and a preset head training feature, and the step of determining the human body region image according to the wearing process image and the preset human body region division basis includes:

[0014] Determine the target image features corresponding to each frame of the wearing process image in sequence;

[0015] When the target image feature matches the torso training feature, determining the wearing process image corresponding to the target image feature as the torso image;

[0016] When the target image feature matches the head training feature, a wearing process image corresponding to the target image feature is determined as a head image.

[0017] In one embodiment, the HRTF implementation method further includes:

[0018] Acquire an input feature learning image, wherein the feature learning image includes a torso feature image and a head feature image;

[0019] Model training is performed based on the trunk feature image and the head feature image to obtain a feature model, and the trunk training features and head training features in the feature model are used as a basis for preset human body region division.

[0020] In one embodiment, the human body region division basis includes a preset wearing process sequence, and after the step of determining the human body region image according to the wearing process image and the preset human body region division basis, the method further includes:

[0021] Determining a target area image in the wearing process image, wherein the target area image includes an ear canal image and a head image;

[0022] An auricle image is determined in the wearing process image based on the wearing process sequence and the ear canal image, and / or an auricle image is determined in the wearing process image based on the wearing process sequence and the head image, and the auricle image is used as a human body area image.

[0023] In one embodiment, the step of obtaining the image information collected by the camera includes:

[0024] Acquiring current status information of the headset terminal, wherein the current status information includes a connection status of the headset terminal and an input control instruction;

[0025] When the connection state is the first connection, a wearing process image of the headset terminal during wearing is collected based on the camera as image information, and / or,

[0026] When the connection state is not the first connection and the control instruction matches the preset acquisition instruction, the wearing process image of the headset terminal during wearing is acquired based on the camera as image information.

[0027] In one embodiment, the human body region image includes a torso image, a head image, an auricle image, and an ear canal image, and the step of determining the target HRTF parameters according to the human body region image includes:

[0028] respectively determining a first HRTF parameter corresponding to the torso image, a second HRTF parameter corresponding to the head image, a third HRTF parameter corresponding to the auricle image, and a fourth HRTF parameter corresponding to the ear canal image;

[0029] A target HRTF parameter is determined based on the first HRTF parameter, the second HRTF parameter, the third HRTF parameter, and the fourth HRTF parameter.

[0030] In one embodiment, the camera includes a depth camera, and the image information includes depth image information.

[0031] In addition, to achieve the above purpose, the present application also provides a HRTF implementation device, which is applied to a headphone terminal that implements HRTF and is equipped with a camera, and the HRTF implementation device includes:

[0032] An image acquisition module, used to acquire image information collected by the camera, wherein the image information includes a wearing process image of the headset terminal during wearing;

[0033] An image division module, used for determining a human body region image according to the wearing process image and a preset human body region division basis;

[0034] The function determination module is used to determine target HRTF parameters according to the human body region image, so as to implement HRTF based on the target HRTF parameters.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides an HRTF implementation device, including a camera, a processor, a memory, and an HRTF implementation method program stored in the memory and executable by the processor, wherein when the HRTF implementation method program is executed by the processor, the steps of the HRTF implementation method described above are implemented.

[0036] The present application also provides a computer-readable storage medium, on which a HRTF implementation method program is stored, wherein when the HRTF implementation method program is executed by a processor, the steps of the HRTF implementation method described above are implemented.

[0037] The embodiment of the present application provides a method for implementing HRTF, which is applied to a headset terminal that implements HRTF and is equipped with a camera, by acquiring image information collected by the camera, wherein the image information includes a wearing process image during the wearing process of the headset terminal; determining a human body region image according to the wearing process image and a preset human body region division basis; determining a target HRTF parameter according to the human body region image, so as to implement HRTF based on the target HRTF parameter, and collecting a wearing process image during the wearing process of the headset terminal by a headset terminal that implements HRTF and is equipped with a camera, and then determining a human body region image based on the wearing process image, and finally determining a target HRTF parameter based on the human body region image to implement HRTF, thereby avoiding the phenomenon that HRTF can only be implemented through an external device and for a static shot image. This HRTF implementation method not only provides a new HRTF implementation method, specifically a new image acquisition method, but also implements HRTF by collecting a wearing process image during the wearing process of the headset terminal by a headset terminal that implements HRTF and is equipped with a camera, thereby avoiding the problem that HRTF can only be implemented using a static shot image, thereby improving the intelligence of HRTF implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the first embodiment of the HRTF implementation method of the present application;

[0039] Figure 2 A physical schematic diagram of an earphone terminal in the HRTF implementation method of the present application;

[0040] Figure 3 A schematic diagram of a wearing process in the HRTF implementation method of the present application;

[0041] Figure 4 A flowchart of the method for implementing HRTF of the present application is shown;

[0042] Figure 5 This is a schematic diagram of the modules of the device for implementing HRTF of this application;

[0043] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in this application.

[0044] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings.

[0045] Description of Figure Numbers:

[0046] 100. earphone terminal; 110. camera; 120. position sensor; 1001. processing device; 1002. ROM; 1003. storage device; 1004. RAM; 1005. bus; 1006. I / O port; 1007. input device; 1008. output device; 1009. communication device; 200. mobile phone. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] At present, the commonly used HRTF implementation methods are mostly based on external devices such as the camera and scanning function of a mobile phone. Personalized HRTF is generated by scanning the left and right ears of a person, or a camera is configured on a head-mounted device such as AR (Augmented Reality) glass to scan the user, and then HRTF is implemented based on the scanned image. This camera and AR glass shooting method can only scan partial images such as torso shape, and cannot fully reflect the individual's physiological characteristics, which results in low accuracy of subsequent HRTF implementation (HRTF is simply performed based on local images). At the same time, the camera and AR glass shooting methods both need to collect static images, so static shooting is required before the entire process, which reduces the intelligence of the entire HRTF implementation.

[0050] Therefore, based on the inadequacies of the above HRTF implementation scheme, the HRTF implementation method of the present application is proposed. The solution of the embodiment of the present application is: by implementing HRTF and configuring a camera to collect the wearing process image of the headphone terminal during wearing, and then determining the human body area image based on the wearing process image, and finally determining the target HRTF parameters based on the human body area image to implement HRTF, thereby avoiding the phenomenon that HRTF can only be implemented through external devices and for static captured images. This HRTF implementation method not only provides a new HRTF implementation method, specifically a new image acquisition method, but also implements HRTF by implementing HRTF and configuring a camera to collect the wearing process image of the headphone terminal during wearing, thereby avoiding the problem that HRTF can only be implemented using static captured images, thereby improving the intelligence of HRTF implementation.

[0051] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a device capable of realizing the above functions, an HRTF implementation device, etc. The following takes the HRTF implementation device as an example to illustrate this embodiment and the following embodiments.

[0052] Based on this, the present application embodiment provides a method for implementing HRTF, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the HRTF implementation method of the present application.

[0053] Reference Figure 1 The present application provides a method for implementing HRTF, which is applied to a headphone terminal that implements HRTF and is equipped with a camera. The method for implementing HRTF includes:

[0054] Step S10, acquiring image information collected by the camera, wherein the image information includes a wearing process image of the headset terminal during wearing;

[0055] For example, see Figure 2 , Figure 2This is a physical schematic diagram of the headphone terminal in the HRTF implementation method of the present application. The headphone terminal 100 in the figure is configured with two or more depth cameras 110 and a position sensor 120. The position sensor 120 can be an optical sensor, a capacitive sensor or a motion sensor (such as an accelerometer / gyroscope), thereby realizing image acquisition during the entire wearing process. It is worth noting that, on the one hand, the headphone terminal 100 can be a component hardware for realizing HRTF, thereby realizing image capture without the use of external mobile phones and other instruments. On the other hand, the controller in the headphone terminal 100 can be the executive device of this embodiment, or other control devices connected to the controller in the headphone terminal 100 as the executive device of this embodiment, which is not limited here. Further, refer to Figure 3 , Figure 3 This is a schematic diagram of a scene of the wearing process in the HRTF implementation method of the present application. The figure takes a Bluetooth headset as an example (of course, it can also be a general smart headset, which is not limited here). The figure sequentially illustrates the process of unpacking the headset terminal 100 and connecting it to the mobile phone 200 through Bluetooth, and then wearing it. At this time, the camera 110 set on the headset terminal 100 can capture the image of the wearing process, which is convenient for subsequent image processing.

[0056] In this embodiment, when the headset terminal equipped with a camera is in the wearing process, the wearing process image of the headset terminal will be captured based on the camera on the headset terminal as image information, wherein the wearing process image refers to the image in the process of wearing the headset. At this time, the image captured by the camera on the headset terminal can be intelligently processed to improve the intelligence of the entire HRTF implementation. It is worth noting that at this time, it is necessary to determine that the image at this time is a wearing process image. The captured image can be used as the wearing process image after sensing the Bluetooth connection, or the subsequent image can be used as the wearing process image if there is a control instruction to capture the image after the Bluetooth connection. Please refer to Figure 4 , Figure 4 This is a flow chart of the implementation method of HRTF of the present application. After the earphone terminal ends the Bluetooth connection, it starts to execute the process of B10 to pick up features. At this time, the user's physiological features are picked up. The picking process is the process image of the user moving the earphone from the front to the ear canal. After the wearing is completed at B20, the final picture picking of B30 will be realized. At this time, the final picture is the process picture from the earphone out of the box to the ear. The time from picking up to wearing is short, so a higher refresh rate is required to pick up and save the user graphics, so as to avoid the common method of using a mobile phone to shoot the user's 3D information, which requires the user to be in a stationary state. It is worth noting that because the earphone terminal is equipped with a depth camera, the user does not need to remain stationary, and the user's earphone wearing process is the process of real-time 3D supplementation of the two earphone terminals.

[0057] Step S20, determining a human body region image according to the wearing process image and a preset human body region division basis;

[0058] In this embodiment, after the wearing process image is determined, the wearing process image will be divided based on the preset human body area division basis to obtain the human body area image, instead of implementing HRTF for an entire image, so as to implement HRTF for different areas in a targeted manner to ensure the accuracy of HRTF implementation. Among them, the preset human body area division basis refers to the basis for the user-defined division image as each different area, and the human body area image refers to the human body image after division, so as to determine the corresponding HRTF parameters based on the human body image after division in a targeted manner to ensure the accuracy of the HRTF parameters finally determined by the device.

[0059] Step S30, determining target HRTF parameters according to the human body region image, so as to implement HRTF based on the target HRTF parameters.

[0060] In this embodiment, after obtaining the human body region image, the target HRTF parameters corresponding to the human body region image are determined, and then personalized HRTF is realized based on the target HRTF parameters. The target HRTF parameters refer to parameters determined based on different human body region images. The parameters can be the sum of parameters determined by different human body region images, the difference of parameters determined by different human body region images, or the convolution sum of parameters determined by different human body region images, etc. For example, the HRTF can be controlled by setting parameters such as interaural time difference, interaural sound level difference, scattering of sound waves by the auricle, and reflected sound of the trunk in the target HRTF parameters. The effect of achieving this is that the parameters of the reflected sound of the torso can be the parameters determined by the torso image, the scattering of the sound wave by the auricle can be the parameters determined by the auricle image, the interaural time difference and the interaural sound level difference can be the parameters determined by the ear canal image, the auricle image and the head image. Taking the interaural time difference as a numerical value as an example, the ear canal image can account for 40% of the numerical value, the auricle image accounts for 40% of the numerical value, and the head image accounts for 20% of the numerical value. If the parameter determined by the ear canal image is A, the parameter determined by the auricle image is B and the parameter determined by the head image is C, then the final interaural time difference is 40%A+40%B+20%C. As shown in reference Figure 4 After wearing is completed, the pictures of the entire wearing process can be determined, such as the torso image, head image, auricle image and ear canal image, and then the process of B40 is executed to generate corresponding physiological parameters based on the images of different regions, and the process of B50 is executed to extract corresponding HRTF parameters based on the characteristics of physiological parameters and the HRTF relationship, and then the HRTF parameters of images in different regions are determined in a targeted manner, that is, personalized HRTF generation, to achieve the effect of personalized HRTF.

[0061] In one embodiment, the camera includes a depth camera, and the image information includes depth image information.

[0062] Exemplarily, the camera includes a depth camera, that is, the image information collected at this time includes depth image information, and the accuracy of the collected image can be guaranteed based on the depth camera, which can be specifically manifested in clarity and other aspects, and the subsequent matching of HRTF parameters can be guaranteed to improve the effect of HRTF. It is worth noting that the camera can also be other cameras, such as a commonly used surveillance camera, which is not limited here.

[0063] In this embodiment, a method for implementing HRTF is provided, which is applied to a headset terminal that implements HRTF and is equipped with a camera, by acquiring image information collected by the camera, wherein the image information includes a wearing process image during the wearing process of the headset terminal; determining a human body region image according to the wearing process image and a preset human body region division basis; determining a target HRTF parameter according to the human body region image, so as to implement HRTF based on the target HRTF parameter, and collecting a wearing process image during the wearing process of the headset terminal by the headset terminal that implements HRTF and is equipped with a camera, and then determining a human body region image based on the wearing process image, and finally determining a target HRTF parameter based on the human body region image to implement HRTF, thereby avoiding the phenomenon that HRTF can only be implemented through an external device and for a static shot image. This method for implementing HRTF not only provides a new way to implement HRTF, specifically a new image acquisition method, but also implements HRTF by collecting a wearing process image during the wearing process of the headset terminal by the headset terminal that implements HRTF and is equipped with a camera, thereby avoiding the problem that HRTF can only be implemented using a static shot image, thereby improving the intelligence of HRTF implementation.

[0064] Further, based on the first embodiment of the present application, a second embodiment of the HRTF implementation method of the present application is proposed. In this embodiment, in the above step S20, the headset terminal includes a position sensor, the human body area image includes an ear canal image, the human body area division basis includes a preset wearing completion state, and the step of determining the human body area image according to the wearing process image and the preset human body area division basis includes:

[0065] Step S201, determining the target position information corresponding to each frame of the wearing process image in sequence, wherein the target position information includes the position state collected by the position sensor;

[0066] Step S202: When the position state matches the wearing completion state, a wearing process image corresponding to the target position information is determined as an ear canal image.

[0067] In this embodiment, the headset terminal includes a position sensor, and the human body region image includes an ear canal image, and then the target position information corresponding to each frame of the wearing process image is determined, that is, the position corresponding to each camera capture image is determined, and the target position information includes the position state captured by the position sensor, that is, the position state at this time, and then when the position state matches the wearing completion state, the wearing process image corresponding to the target position information is determined as the ear canal image, and the wearing completion state refers to the user-defined detection state in the ear canal. For example, the auricle and ear canal image picking process are linked with the wearing detection function, and the optical sensor, capacitive sensor or motion sensor (such as accelerometer / gyroscope) with the wearing function of the headset terminal can detect the movement and position change of the headset. Taking the infrared sensor as an example, when the headset terminal is worn, the infrared beam is emitted from the transmitter and reflected by the tissue in the ear canal. The receiver detects the reflected light, and it is considered that the wearing is completed at this time. At this time, the photo taken by the camera in the previous frame is considered to be the auricle information. Based on the above method, the ear canal image can be quickly distinguished, so that the HRTF parameters of the ear canal image can be determined in a targeted manner later, thereby ensuring the accuracy of the ear canal HRTF.

[0068] Further, based on the first embodiment and / or the second embodiment of the present application, a third embodiment of the method for implementing HRTF of the present application is proposed. In this embodiment, in the above step S20, the human body region image includes a torso image and a head image, and the human body region division basis includes a preset torso training feature and a preset head training feature. The step of determining the human body region image according to the wearing process image and the preset human body region division basis includes:

[0069] Step S211, determining the target image features corresponding to each frame of the wearing process image in turn;

[0070] Step S212, when the target image feature matches the torso training feature, determining the wearing process image corresponding to the target image feature as the torso image;

[0071] Step S213, when the target image feature matches the head training feature, determining the wearing process image corresponding to the target image feature as the head image.

[0072] In this embodiment, the human body region image includes a torso image and a head image, and the human body region division basis includes a preset torso training feature and a preset head training feature, the torso training feature refers to the torso feature determined based on the relevant model training or the neural network, and the head training feature refers to the head feature determined based on the relevant model training or the neural network. Then, the target image feature corresponding to each frame of the wearing process image is determined, and the target image feature is respectively compared with the torso training feature and the head training feature to determine which region of the wearing process image the target image feature corresponds to at this time. The target image feature refers to the image feature corresponding to a certain frame of the wearing process image, and the torso image and the head image are determined by the target image feature, so as to facilitate subsequent targeted processing.

[0073] Furthermore, the HRTF implementation method also includes:

[0074] Step S01, obtaining an input feature learning image, wherein the feature learning image includes a torso feature image and a head feature image;

[0075] Step S02, performing model training based on the trunk feature image and the head feature image to obtain a feature model, and using the trunk training features and the head training features in the feature model as a basis for preset human body region division.

[0076] In this embodiment, before the implementation method of the entire HRTF is executed, a deep neural network is also trained to identify the head and torso in the image, that is, through training with a large amount of labeled data, the model can learn the complex features that distinguish the head and torso. The input feature learning image can be obtained, wherein the feature learning image includes a torso feature image and a head feature image, and then the model is trained based on the torso feature image and the head feature image to obtain a feature model, and the torso training features and head training features in the feature model are used as the basis for the preset human body area division, so as to quickly and accurately determine the torso image and head image in the entire wearing process image later. Among them, the torso feature image refers to the image related to the torso, the head feature image refers to the image related to the head, and the feature model refers to the related feature model of the torso and head obtained after training, and then the torso training features and head training features are used as the basis for subsequent judgment of the torso image and the head image.

[0077] In one embodiment, because the entire wearing process image also includes the auricle image, the auricle image can also be trained to obtain the auricle training features, that is, the auricle features determined based on relevant model training or neural network, and then the auricle image can be determined in the wearing process image based on the auricle training features.

[0078] Further, based on the first embodiment, the second embodiment and / or the third embodiment of the present application, a fourth embodiment of the method for implementing HRTF of the present application is proposed. In this embodiment, in the above step S20, the human body area division basis includes a preset wearing process sequence, and after the step of determining the human body area image according to the wearing process image and the preset human body area division basis, the method includes:

[0079] Step S221, determining a target area image in the wearing process image, wherein the target area image includes an ear canal image and a head image;

[0080] Step S222, determining the auricle image in the wearing process image based on the wearing process sequence and the ear canal image, and / or determining the auricle image in the wearing process image based on the wearing process sequence and the head image, and using the auricle image as the human body area image.

[0081] In this embodiment, the auricle image can also be determined after the ear canal image and the head image are determined. At this time, the human body area division is based on a preset wearing process sequence, and the wearing process sequence refers to the movement sequence of the earphone terminal during the wearing process. By determining the target area image in the wearing process image, wherein the target area image includes the ear canal image and the head image, the auricle image can be determined in the wearing process image based on the shooting sequence of the wearing process image, the wearing process sequence and the ear canal image, or the auricle image can be determined in the wearing process image based on the shooting sequence of the wearing process image, the wearing process sequence and the head image, that is, the relationship between the front and back frames of the shot image is determined, and the ear canal image and the head image can be determined to determine the auricle image. When the two methods are inconsistent, the human body area image of the entire wearing process image can be re-determined, thereby ensuring the accuracy of the area determination, so as to ensure the accuracy of the subsequent HRTF.

[0082] In one embodiment, the step of obtaining the image information collected by the camera includes:

[0083] Step S11, obtaining current status information of the headset terminal, wherein the current status information includes a connection status of the headset terminal and an input control instruction;

[0084] Step S12, when the connection state is the first connection, collecting a wearing process image of the headset terminal during wearing as image information based on the camera, and / or,

[0085] Step S13, when the connection state is not the first connection and the control instruction matches the preset collection instruction, the wearing process image of the headset terminal during wearing is collected based on the camera as image information.

[0086] In this embodiment, when collecting image information, the current state information of the headset terminal is obtained, and then based on the current state information, it is determined whether the image collected at this time needs to be used as image information. The connection state of the headset terminal refers to the Bluetooth connection state of the headset terminal, such as connected, first connection, etc. The control instruction refers to the control instruction input to the headset terminal, such as control shutdown, control image acquisition, etc. Then, when the connection state is the first connection, the wearing process image of the headset terminal during wearing is collected based on the camera as image information, and / or, when the connection state is not the first connection, and the control instruction matches the preset acquisition instruction, the wearing process image of the headset terminal during wearing is collected based on the camera as image information, that is, when there is a need for image acquisition or the first Bluetooth connection is met, at least one of the conditions is met, and the image at this time can be obtained as image information. It is worth noting that at this time, the sensor in the headset terminal can be used to detect whether the headset terminal is in a mobile state, so as to wake up the camera when it is in a mobile state to collect the wearing process image of the headset terminal during wearing. For example, when the headset terminal is used for the first time, the headset terminal is generally placed in the headset box to connect to Bluetooth, and then worn after the Bluetooth connection is completed. Therefore, after the first Bluetooth connection, the personalized wearing process image acquisition can be automatically entered; if the current state of the headset terminal and the mobile phone have been paired with Bluetooth, you can choose whether to perform personalized wearing process image acquisition on the interface. If personalized acquisition is not performed (that is, after executing steps S10-S30 to realize image acquisition, the new target HRTF parameters are re-determined for HRTF control), the system will use the personalized wearing process image information collected by the original wearer. If personalized wearing process image acquisition is selected, the headset terminal needs to be put back from the headset to the headset box, and the headset terminal is taken out to start personalized wearing process image acquisition. It is worth noting that the headset terminal can retain multiple personalized wearing process images, or only one personalized data, so that the historical personalized wearing process images can be used quickly, thereby greatly improving the implementation efficiency of HRTF.

[0087] Further, based on the first embodiment, the second embodiment, the third embodiment and / or the fourth embodiment of the present application, a fifth embodiment of the method for implementing HRTF of the present application is proposed. In this embodiment, the human body region image includes a torso image, a head image, an auricle image and an ear canal image. The step of determining the target HRTF parameters according to the human body region image includes:

[0088] Step a, respectively determining a first HRTF parameter corresponding to the torso image, a second HRTF parameter corresponding to the head image, a third HRTF parameter corresponding to the auricle image, and a fourth HRTF parameter corresponding to the ear canal image;

[0089] Step b: determining target HRTF parameters based on the first HRTF parameters, the second HRTF parameters, the third HRTF parameters and the fourth HRTF parameters.

[0090] In this embodiment, the human body region image includes a torso image, a head image, an auricle image and an ear canal image, and then a personalized HRTF is performed based on the torso image, the head image, the auricle image and the ear canal image. On the one hand, the torso image is directly matched with the relevant first HRTF parameters, and at the same time, the second HRTF parameters corresponding to the head image, the third HRTF parameters corresponding to the auricle image and the fourth HRTF parameters corresponding to the ear canal image are determined in combination with the deep learning algorithm, so as to realize a truly personalized HRTF and achieve a more natural spatial audio rendering. Final picture pickup: After wearing, the final state image is picked up on the L / R earphones. Among them, the first HRTF parameter is the HRTF parameter corresponding to the torso image, which can generally be matched in the HRTF parameter library, the second HRTF parameter is the HRTF parameter corresponding to the head image, which generally needs to be trained to obtain personalized parameters, the third HRTF parameter is the HRTF parameter corresponding to the auricle image, which generally needs to be trained to obtain personalized parameters, and the fourth HRTF parameter is the HRTF parameter corresponding to the ear canal image, which generally needs to be trained to obtain personalized parameters. Finally, the target HRTF parameter can be determined based on the first HRTF parameter, the second HRTF parameter, the third HRTF parameter and the fourth HRTF parameter, and the determination method can be to average the four parameters, or to convolve the four parameters. Because the HRTF parameters corresponding to the head image, the auricle image and the ear canal image have a small matching relationship with each other, and are directly related to the final effect, the HRTF parameters corresponding to the head image, the auricle image and the ear canal image are trained to ensure personalized HRTF implementation. At the same time, the acquisition is performed based on the acquisition method of the headset terminal, and the HRTF parameters are determined by intelligently dividing into multiple areas, respectively, to ensure the intelligence and accuracy of the entire HRTF implementation.

[0091] In one embodiment, the commonly used HRTF implementation method is generally to find similar ones from the HRTF library. If the amount of database data is small, the similar HRTF found will have a poor match and low personalization to a large extent. Or complete deep learning requires a lot of computing power to achieve HRTF adaptation. Therefore, it is proposed in this embodiment to divide the head, torso, auricle and ear canal into images that can be directly matched and those that need to be trained, thereby improving the personalization of HRTF. When matching the head, torso, auricle and ear canal with the HRTF library, a series of physiological parameters are usually used for matching. And use these parameters for database matching to achieve personalization of HRTF. These parameters include but are not limited to: head size: such as the length, width and height of the head; auricle size: including the distance from the auricle to the head, the length and width of the auricle; inter-ear distance: the distance between the two ears, which is a key factor affecting the difference in the time it takes for sound to reach each ear; facial size: such as the distance from the eyes to the ears; torso size: the shape and size of the torso will also affect the propagation of sound. Then, matching or training can be performed based on the above parameters to ensure the personalization of HRTF. It is worth noting that, referring to Figure 4 After determining the human body area image, the physiological parameters will be determined based on the three-dimensional data of the human body area image, including size information (length, width, height, ear size and shape, etc.), geometric parameters (curvature, angle, area, volume, etc. of the surface), and morphological parameters (such as the shape characteristics of the auricle, head, and ear canal, etc.). Finally, it can be extracted based on the determined physiological parameters and HRTF relationship: This function is based on a huge personalized HRTF library (the HRTF library consists of four parts, the trunk class Ha, the head class Hb, the auricle class Hc and the ear canal class Hd, then a user's HRTF is H = Ha * Hb * Hc * Hd). In particular, in order to reduce computing power, first search for completely consistent parameter matches in the trunk, head, ear canal and auricle categories. Often the trunk part can match the completely consistent Has (because the trunk size is easily affected by the size and style of clothes, at this time Has represents the parameters matched by the user in the Ha library), while Hb, Hc and Hd are almost difficult to match due to their individual uniqueness. In order to increase the accuracy of personalized HRTF, Hb, Hc and Hd are recreated in this dimension through deep learning to obtain Hbn, Hcn and Hdn for the user. At the same time, Hbn, Hcn and Hdn are uploaded to the cloud library to supplement the HRTF library. Finally, personalized HRTF generation is achieved: at this time, the HRTF of the user is H = Has*Hbn*Hcn*Hdn. Finally, personalized HRTF can be achieved based on H to ensure the implementation effect of HRTF.

[0092] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the implementation method of the HRTF of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0093] This application also provides a HRTF implementation device, please refer to Figure 5 The HRTF implementation device is applied to a headphone terminal that implements HRTF and is equipped with a camera, and the HRTF implementation device includes:

[0094] An image acquisition module A10 is used to acquire image information collected by the camera, wherein the image information includes a wearing process image of the headset terminal during wearing;

[0095] An image division module A20, configured to determine a human body region image according to the wearing process image and a preset human body region division basis;

[0096] The function determination module A30 is used to determine target HRTF parameters according to the human body region image, so as to implement HRTF based on the target HRTF parameters.

[0097] The HRTF implementation device provided in this application adopts the HRTF implementation method in the above-mentioned embodiment, which can solve the technical problem of low intelligence of HRTF implementation. Compared with the prior art, the beneficial effects of the HRTF implementation device provided in this application are the same as the beneficial effects of the HRTF implementation method provided in the above-mentioned embodiment, and other technical features in the HRTF implementation device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0098] The present application provides a HRTF implementation device (which can be a headphone terminal or a virtual device), the HRTF implementation device includes: a camera, at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the HRTF implementation method in the above-mentioned embodiment one.

[0099] Exemplarily, the HRTF implementation device can be various devices equipped with cameras, which can be Bluetooth headsets or conventional smart headsets, etc. In this case, cameras can be set in the two headsets to capture images of the wearing process of the headsets. At the same time, in order to ensure parameters such as the clarity of the captured image, depth cameras can be set in the two headsets respectively to realize image capture.

[0100] Reference below Figure 6, which shows a schematic diagram of the structure of an implementation device of HRTF suitable for implementing the embodiment of the present application. The implementation device of HRTF in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The HRTF implementation device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0101] like Figure 6 As shown, the HRTF implementation device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the HRTF implementation device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following devices can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the HRTF implementation device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows an HRTF implementation device with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0102] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0103] The HRTF implementation device provided by the present application adopts the HRTF implementation method in the above-mentioned embodiment, which can solve the technical problem of low intelligence of HRTF implementation. Compared with the prior art, the beneficial effects of the HRTF implementation device provided by the present application are the same as the beneficial effects of the HRTF implementation method provided by the above-mentioned embodiment, and other technical features in the HRTF implementation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0104] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0105] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0106] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the HRTF implementation method in the above-mentioned embodiment.

[0107] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, apparatuses or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution device, apparatus or component. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0108] The computer-readable storage medium may be included in the HRTF implementation device; or may exist independently without being assembled into the HRTF implementation device.

[0109] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the HRTF implementation device, the HRTF implementation device:

[0110] Acquire image information collected by the camera, wherein the image information includes a wearing process image of the headset terminal during wearing;

[0111] Determine a human body region image according to the wearing process image and a preset human body region division basis;

[0112] The target HRTF parameters are determined according to the human body region image to implement the HRTF based on the target HRTF parameters.

[0113] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0116] The storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned HRTF implementation method, and can solve the technical problem that the HRTF implementation is not intelligent enough. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the HRTF implementation method provided in the above-mentioned embodiment, and will not be described in detail here.

[0117] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned HRTF implementation method when executed by a processor.

[0118] The computer program product provided by this application can solve the technical problem that the HRTF implementation is not intelligent enough. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the HRTF implementation method provided by the above embodiment, which will not be repeated here.

[0119] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for implementing HRTF, characterized in that: The HRTF implementation method is applied to a headphone terminal that implements HRTF and is equipped with a camera, and the HRTF implementation method includes: Acquire image information collected by the camera, wherein the image information includes a wearing process image of the headset terminal during wearing; Determine a human body region image according to the wearing process image and a preset human body region division basis; Target HRTF parameters are determined according to the human body region image to implement HRTF based on the target HRTF parameters.

2. The HRTF implementation method according to claim 1, characterized in that: The headset terminal includes a position sensor, the human body region image includes an ear canal image, the human body region division basis includes a preset wearing completion state, and the step of determining the human body region image according to the wearing process image and the preset human body region division basis includes: Determine the target position information corresponding to each frame of the wearing process image in sequence, wherein the target position information includes the position state collected by the position sensor; When the position state matches the wearing completion state, a wearing process image corresponding to the target position information is determined as an ear canal image.

3. The method for implementing HRTF according to claim 1, characterized in that: The human body region image includes a torso image and a head image, the human body region division basis includes a preset torso training feature and a preset head training feature, and the step of determining the human body region image according to the wearing process image and the preset human body region division basis includes: Determine the target image features corresponding to each frame of the wearing process image in sequence; When the target image feature matches the torso training feature, determining the wearing process image corresponding to the target image feature as the torso image; When the target image feature matches the head training feature, a wearing process image corresponding to the target image feature is determined as a head image.

4. The method for implementing HRTF according to claim 3, characterized in that: The HRTF implementation method further includes: Acquire an input feature learning image, wherein the feature learning image includes a torso feature image and a head feature image; Model training is performed based on the trunk feature image and the head feature image to obtain a feature model, and the trunk training features and head training features in the feature model are used as a basis for preset human body region division.

5. The method for implementing HRTF according to claim 1, characterized in that: The human body region division basis includes a preset wearing process sequence, and after the step of determining the human body region image according to the wearing process image and the preset human body region division basis, the method further includes: Determining a target area image in the wearing process image, wherein the target area image includes an ear canal image and a head image; An auricle image is determined in the wearing process image based on the wearing process sequence and the ear canal image, and / or an auricle image is determined in the wearing process image based on the wearing process sequence and the head image, and the auricle image is used as a human body area image.

6. The method for implementing HRTF according to claim 1, characterized in that: The step of obtaining the image information collected by the camera includes: Acquiring current status information of the headset terminal, wherein the current status information includes a connection status of the headset terminal and an input control instruction; When the connection state is the first connection, a wearing process image of the headset terminal during wearing is collected based on the camera as image information, and / or, When the connection state is not the first connection and the control instruction matches the preset acquisition instruction, the wearing process image of the headset terminal during wearing is acquired based on the camera as image information.

7. The method for implementing HRTF according to any one of claims 1 to 6, characterized in that: The human body region image includes a torso image, a head image, an auricle image, and an ear canal image. The step of determining the target HRTF parameters according to the human body region image includes: respectively determining a first HRTF parameter corresponding to the torso image, a second HRTF parameter corresponding to the head image, a third HRTF parameter corresponding to the auricle image, and a fourth HRTF parameter corresponding to the ear canal image; A target HRTF parameter is determined based on the first HRTF parameter, the second HRTF parameter, the third HRTF parameter, and the fourth HRTF parameter.

8. The method for implementing HRTF according to any one of claims 1 to 6, characterized in that: The camera includes a depth camera, and the image information includes depth image information.

9. A device for implementing HRTF, characterized in that: The HRTF implementation device is applied to a headphone terminal that implements HRTF and is equipped with a camera, and the HRTF implementation device includes: An image acquisition module, used to acquire image information collected by the camera, wherein the image information includes a wearing process image of the headset terminal during wearing; An image division module, used for determining a human body region image according to the wearing process image and a preset human body region division basis; The function determination module is used to determine target HRTF parameters according to the human body region image, so as to implement HRTF based on the target HRTF parameters.

10. A device for implementing HRTF, characterized in that: The HRTF implementation device includes a camera, a processor, a memory, and an HRTF implementation method program stored in the memory and executable by the processor, wherein when the HRTF implementation method program is executed by the processor, the steps of the HRTF implementation method as described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for implementing a method for HRTF, wherein when the program for implementing a method for HRTF is executed by a processor, the steps of the method for implementing HRTF as claimed in any one of claims 1 to 8 are implemented.

Images