Sound source signal processing method and device, display equipment and storage medium

Abstract

The embodiment of the invention relates to a sound source signal processing method and device, display equipment and a storage medium, and the method comprises the steps: determining a plurality of pieces of position information of a target object in the display equipment when the moving target object exists in the display equipment; obtaining a filter corresponding to each piece of position information, wherein each piece of position information corresponds to one filter; and in the moving process of the target object, superposing the sound source signal with the filter corresponding to each piece of position information so as to output a superposed virtual sound image signal through a loudspeaker of the display equipment. Therefore, the virtual sound image signal can be generated according to the position of the moving target object in the display equipment, so that sound heard by a user is matched with the position of the target object on the display equipment, the position of the virtual sound image signal coincides with the moving object on the display equipment, and the obtained position of the virtual sound image signal is more accurate; and the accuracy of sound and picture tracking and the audio-visual experience of the user are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of sound source processing, and in particular to a sound source signal processing method, apparatus, display device and storage medium. Background Art

[0002] When we watch TV, the objects on the screen are usually moving and their positions are changing. For example, a flying bird may be located in the lower left corner of the screen at the moment, and may fly to the upper right corner of the screen the next moment. However, the positions of the speakers configured for the TV remain unchanged, so the positions of the sound and image produced by each speaker remain unchanged. For example, if the speakers are installed symmetrically on the left and right sides under the TV, the sound and image are always in the middle of the TV. This results in the sound and image positions being inconsistent with the positions of the objects in the picture, which is different from the feeling in real life, and brings a relatively poor viewing experience to users.

[0003] Some methods for achieving sound and image tracking have also been proposed in the prior art. Speakers are arranged at the bottom and top of the TV, and the amplitude and phase of each speaker are controlled to make the synthetic sound and image position coincide with the position of the moving object. However, most of the TVs we use are still dual-channel, with only one speaker installed on each of the left and right sides of the bottom of the TV, and no speaker on the top. At this time, the sound field is basically only distributed in the bottom area of ​​the TV. In this case, the method of controlling the amplitude and phase of each speaker in the existing technology cannot make the sound and image position at the top of the screen. When the bird in the TV picture flies to the top of the screen, the sound and image position cannot coincide with the current bird position. Therefore, how to improve the accuracy of sound and image tracking to improve the user's audio-visual experience has become a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, in order to solve the above technical problems or part of the technical problems, the embodiments of the present invention provide a sound source signal processing method, apparatus, display device and storage medium.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing an audio source signal, comprising:

[0006] When there is a moving target object in the display device, determining a plurality of position information of the target object in the display device;

[0007] Obtain a filter corresponding to each piece of the position information, wherein each piece of the position information corresponds to a filter;

[0008] During the movement of the target object, the sound source signal is superimposed on the filter corresponding to each of the position information, so as to output the superimposed virtual sound and image signal through the speaker of the display device.

[0009] In a possible implementation manner, determining a plurality of position information of the target object in the display device includes:

[0010] Acquire a target movement trajectory of the target object in the display device;

[0011] Acquire multiple grid areas of the display device;

[0012] Determine a preset number of target grid areas from the grid areas through which the target moving trajectory passes;

[0013] Each target grid area is determined as a piece of position information, and a plurality of pieces of position information are obtained.

[0014] In a possible implementation, during the movement of the target object, superimposing the sound source signal with the filter corresponding to each piece of the position information includes:

[0015] Acquiring real-time location information of the target object during its movement;

[0016] Determine the location information closest to the real-time location information as the target location information;

[0017] Superimposing the sound source signal with the filter corresponding to the target position information to obtain a new sound source signal;

[0018] The new sound source signal is subjected to crosstalk elimination processing to obtain the virtual sound and image signal.

[0019] In a possible implementation manner, performing crosstalk elimination processing on the new audio source signal includes:

[0020] Obtaining a system transfer function frequency response matrix from each of the loudspeakers to the left ear and the right ear of a human body at a viewing position corresponding to the display device;

[0021] Performing an inverse operation on the frequency response matrix of the system transfer function to obtain a frequency response matrix of the crosstalk cancellation filter;

[0022] The new audio source signal is subjected to crosstalk cancellation processing according to the frequency response matrix of the crosstalk cancellation filter.

[0023] In a possible implementation manner, before determining a plurality of position information of the target object in the display device, the method further includes:

[0024] Calculating a head-related transfer function from each grid area in the display device to a viewing position corresponding to the display device as a filter for each grid area;

[0025] storing the viewing position and the filter of each grid area in a database;

[0026] The obtaining of the filter corresponding to each piece of the position information includes:

[0027] A filter corresponding to each piece of the position information is obtained from the database.

[0028] In a possible implementation, the method further includes:

[0029] When the viewing position changes, a head-related transfer function from each grid area to the new viewing position is calculated as a new filter;

[0030] The database is updated according to the new filter and the new viewing position.

[0031] In a possible implementation, the method further includes:

[0032] When there are multiple target objects in the display device, determining a first target object and a second target object from the multiple target objects according to the importance of each target object;

[0033] Determine first position information of the first target object and second position information of the second target object;

[0034] During the movement of the first target object, superimposing a first sound source signal corresponding to the first target object and a filter corresponding to the first position information, so as to output a superimposed first virtual sound and image signal through a first speaker of the display device;

[0035] During the movement of the second target object, the second sound source signal corresponding to the second target object is superimposed on the filter corresponding to the second position information, so as to output the superimposed second virtual sound and image signal through the second speaker of the display device.

[0036] In a second aspect, an embodiment of the present invention provides a sound source signal processing device, comprising:

[0037] When there is a moving target object in the display device, determining a plurality of position information of the target object in the display device;

[0038] Obtain a filter corresponding to each piece of the position information, wherein each piece of the position information corresponds to a filter;

[0039] During the movement of the target object, the sound source signal is superimposed on the filter corresponding to each of the position information, so as to output the superimposed virtual sound and image signal through the speaker of the display device.

[0040] In a third aspect, an embodiment of the present invention provides a display device, comprising: a processor and a memory, wherein the processor is used to execute a sound source signal processing program stored in the memory to implement the sound source signal processing method described in any one of the first aspects above.

[0041] In a fourth aspect, an embodiment of the present invention provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the sound source signal processing method described in any one of the first aspects above.

[0042] The sound source signal processing solution provided by the embodiment of the present invention determines multiple position information of the target object in the display device when there is a moving target object in the display device; obtains the filter corresponding to each of the position information, and each of the position information corresponds to a filter; during the movement of the target object, the sound source signal is superimposed with the filter corresponding to each of the position information, so as to output the superimposed virtual sound and image signal through the speaker of the display device. Thus, a virtual sound and image signal can be generated by the position of the moving target object in the display device, so that the sound heard by the user matches the position of the target object on the display device, so that the position of the virtual sound and image signal coincides with the moving object on the display device, so that the position of the virtual sound and image signal is more accurate, and the accuracy of sound and image tracking and the user's audio-visual experience are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a method for processing a sound source signal provided by an embodiment of the present invention;

[0044] Figure 2 A schematic flow chart of another method for processing a sound source signal provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of the structure of a sound source signal processing device provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the structure of a display device provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of a scenario for generating a database provided by an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of display content of a display device provided by an embodiment of the present invention;

[0049] Figure 7 A schematic flow chart of another method for processing a sound source signal provided by an embodiment of the present invention;

[0050] Figure 8 A schematic flow chart of another method for processing a sound source signal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0053] Figure 1 A flow chart of a method for processing a sound source signal provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the method specifically includes:

[0054] S11. When there is a moving target object in the display device, determine multiple position information of the target object in the display device.

[0055] The method for processing a sound source signal provided by an embodiment of the present invention is applied to a display device, which has a speaker and may include but is not limited to: a smart TV, a projection device, a tablet computer, etc. Specifically, the corresponding filter is determined by obtaining the position of a target object moving in the display device, and the filter is superimposed on the sound source signal so that the speaker outputs the superimposed virtual sound and image signal.

[0056] In this embodiment, when the display device plays a video source, a target object that is moving in the screen of the display device and corresponds to the current sound source is obtained. For example, if the current sound source is an airplane flying by, and the display device includes a moving airplane, the airplane is determined to be the target object, and all movement trajectories of the target object in the video source in the screen of the display device are obtained, and multiple positions that the target object passes through in the screen of the display device are selected from the movement trajectories as position information.

[0057] S12: Obtain a filter corresponding to each piece of the position information, where each piece of the position information corresponds to a filter.

[0058] In this embodiment, a head-related transfer function (HRTF) is pre-set for each position in the screen of the display device as a filter from the position to the user's viewing position. HRTF is a sound localization algorithm. The signal transmitted from any point in space to the human ear (in front of the eardrum) can be described by a filter system. The sound source superposition filter obtains the sound signal in front of the eardrums of both ears. If we obtain this set of filters (transfer functions) that describe spatial information, namely HRTF, we can restore the sound signal from each position in the display device.

[0059] Specifically, the binaural filters of the human body from each position of the display device screen to the preset viewing position of the display device are calculated in advance through HRTF, and a filter group from all positions to the binaurals is obtained as a filter matrix for storage.

[0060] After multiple pieces of position information of the current target object are determined, a filter corresponding to each piece of position information is obtained.

[0061] S13. During the movement of the target object, the sound source signal is superimposed with the filter corresponding to each of the position information, so as to output the superimposed virtual sound and image signal through the speaker of the display device.

[0062] In this embodiment, the target object will move to different position information when the video is played. When it is detected that the target object has moved to a certain position information, the sound source signal corresponding to the current position information is superimposed with the corresponding filter to generate a virtual sound and image signal. The virtual sound and image signal is output through the speaker corresponding to the display device. When the position information changes, the corresponding sound source signal and filter also change accordingly. Thus, the corresponding virtual sound and image signal can be output according to different position information, so that the sound heard by the user moves with the moving position of the target object on the display device.

[0063] The sound source signal processing method provided by the embodiment of the present invention determines multiple position information of the target object in the display device when there is a moving target object in the display device; obtains the filter corresponding to each of the position information, and each of the position information corresponds to a filter; during the movement of the target object, the sound source signal is superimposed with the filter corresponding to each of the position information, so as to output the superimposed virtual sound and image signal through the speaker of the display device. Thus, a virtual sound and image signal can be generated by the position of the moving target object in the display device, so that the sound heard by the user matches the position of the target object on the display device, so that the position of the virtual sound and image signal coincides with the moving object on the display device, so that the position of the virtual sound and image signal is more accurate, and the accuracy of sound and image tracking and the user's audio-visual experience are improved.

[0064] Figure 2 A flow chart of another method for processing a sound source signal provided by an embodiment of the present invention is shown as follows: Figure 2 As shown, the method specifically includes:

[0065] S21, calculating the head-related transfer function from each grid area in the display device to the viewing position corresponding to the display device as a filter for each grid area; and storing the viewing position and the filter for each grid area in a database.

[0066] In this embodiment, a plurality of parallel and vertical lines are pre-set in the screen of the display device to divide the screen into a plurality of grid areas. The size and number of the grid areas can be set according to requirements. The larger the number, the more accurate the position of the virtual audio-visual signal generation is. A viewing position in front of the display area is pre-set. The viewing position is the position where the user usually views the display device (for example, three meters directly in front of the display device). An artificial head is placed at the viewing position, and a head-related transfer function from each grid area in the display device to the artificial head at the viewing position is calculated as a filter corresponding to the grid area at the position. Specifically, a special microphone can be placed in the left ear canal and the right ear canal of the artificial head respectively. The audio frequency response of the sound source signal is flat. When the sound source makes a sound at a certain position, the audio frequency response obtained by the microphone recording is no longer flat, which can be equivalent to adding a filter processing. The line connecting the center point of each grid area of ​​the display device and the center point of the artificial head will form different angles with the line connecting the center point of the display device screen and the center point of the artificial head. A set of filters can be generated corresponding to each angle.

[0067] When the sound source is set at different positions of the display device, the sound frequency response obtained by the microphone recording is different. By calculating the head-related transfer function of the audio obtained by the two microphones when the sound source is sounded at the position of each grid area of ​​the display device, as the filter corresponding to each grid area, the filter matrix of the display device relative to the current viewing position is obtained. Similarly, if Figure 5 The figure shows a scenario diagram of generating a database provided by an embodiment of the present invention. If it is expanded to a three-dimensional sphere, the filter matrix can be obtained by recording the microphone recording data when the sound source signal is located at different positions on the sphere.

[0068] Furthermore, different viewing positions can be adjusted to obtain filters corresponding to different grid areas under multiple viewing positions, and the corresponding relationship between the current viewing position, different grid areas, filters, and filter matrices is stored in a database, wherein each viewing position corresponds to a filter matrix, each filter matrix contains multiple filters, and each filter corresponds to a grid area.

[0069] In a possible implementation, when the viewing position changes, the head-related transfer function from each grid area to the new viewing position is calculated as a new filter; and a new correspondence is generated according to the new filter and the new viewing position to update the database.

[0070] S22, obtaining a target movement trajectory of the target object in the display device; obtaining multiple grid areas of the display device; determining a preset number of target grid areas from the grid areas passed by the target movement trajectory; determining each target grid area as a position information, and obtaining multiple position information.

[0071] In this embodiment, a video source to be played on a display device is obtained, a moving target object is obtained from the video source, and a target movement trajectory of the target object in the display device is obtained. The target movement trajectory is obtained passing through multiple grid areas. Since the target movement trajectory may pass through more grid areas, a preset number (for example, 3) of target grid areas are determined from the multiple grid areas. Usually, the positions of the target grid areas in the display device are usually dispersed. For example, the straight-line distance between the determined multiple target grid areas can be set to be greater than a set value.

[0072] Further, each determined target grid area is determined as a piece of position information, and a plurality of position information are obtained.

[0073] S23. Obtain a filter corresponding to each piece of position information, where each piece of position information corresponds to one filter.

[0074] In this embodiment, the filter corresponding to each position information is obtained from a pre-generated database. Since the database is stored in a corresponding relationship, the current position of the user's head in front of the display device is first determined, the viewing position closest to the position is determined as the target viewing position, the filter matrix corresponding to the target viewing position is determined from the database, and the filter corresponding to the grid area where each current position information is located is determined from the filter matrix.

[0075] S24, obtaining the real-time position information of the target object during its movement; determining the position information closest to the real-time position information as the target position information; superimposing the sound source signal with the filter corresponding to the target position information to obtain a new sound source signal; performing crosstalk elimination processing on the new sound source signal to obtain a virtual sound and image signal.

[0076] In this embodiment, the real-time position information of the target object is obtained in real time during the movement of the target object in the screen of the display device, the distance between the real-time position information and each position information is determined, and the position information closest to the target position information is determined as the target position information, which represents the position where the virtual audio and video signal needs to be generated at present. When the real-time position information changes, the target position information is updated at the same time.

[0077] Furthermore, the filter corresponding to the grid area where the current target position information is located is determined, and the sound source signal is superimposed on the filter to obtain a new sound source signal. Since the two speakers configured in the display device produce sound at the same time, there will be mutual interference, and there is no sound interference in the HRTF database collection process. Therefore, for a two-channel display device, the generated new sound source signal needs to be crosstalk-eliminated to obtain the final virtual sound and image signal.

[0078] Specifically, the crosstalk elimination process is performed on the new audio source signal, including:

[0079] Obtain the system transfer function frequency response matrix of the left ear and the right ear of the human body at the viewing position corresponding to each loudspeaker to the display device; perform an inverse operation on the system transfer function frequency response matrix to obtain the frequency response matrix of the crosstalk elimination filter; and perform crosstalk elimination processing on the new sound source signal according to the frequency response matrix of the crosstalk elimination filter.

[0080] In this embodiment, it is necessary to generate sound information containing only the channels corresponding to the binaural sound signals at the entrances of the binaural ear canals of the user at the viewing position, that is, the sound of the left channel speaker only enters the user's left ear, and the sound of the right channel speaker only enters the user's right ear, so that the user can accurately obtain the binaural information in the binaural sound recording, and then generate realistic virtual sound and image signals. The frequency response matrix of the crosstalk cancellation filter can be predetermined based on the crosstalk cancellation method of inverting the system transfer function matrix. The specific process of the signal from the speaker to the human ear can be expressed as:

[0081] P=CHd

[0082] In the formula is the sound pressure vector received by the human ear, is the frequency response matrix of the system transfer function from two speakers to two ears, is the frequency response matrix of the crosstalk cancellation filter, is the spectrum of the signal to be played back. In order to make the signal received by the human ear consistent with the playback signal, the following conditions must be met:

[0083] P=CHd=d

[0084] CH=I

[0085] H=C -1

[0086] Among them, I is the unit matrix. The measurement of the frequency response matrix of the system transfer function is obtained by performing Wiener filtering matching on the signal of the input speaker and the signal collected by the artificial head at the viewing position. The frequency response matrix of the crosstalk elimination filter is obtained by inverting it. The crosstalk elimination operation can be performed based on the frequency response matrix of the crosstalk elimination filter so that each speaker outputs a virtual sound and image signal.

[0087] In one possible implementation, when there are multiple target objects in a display device, a first target object and a second target object are determined from the multiple target objects according to the importance of each target object; first position information of the first target object and second position information of the second target object are determined; during the movement of the first target object, a first sound source signal corresponding to the first target object is superimposed with a filter corresponding to the first position information, so as to output a superimposed first virtual sound and image signal through a first speaker of the display device; during the movement of the second target object, a second sound source signal corresponding to the second target object is superimposed with a filter corresponding to the second position information, so as to output a superimposed second virtual sound and image signal through a second speaker of the display device.

[0088] Specifically, the importance of different types of objects is pre-set and sorted. When there are multiple target objects in the picture at the same time, since there are only two speakers, it is impossible to output a virtual sound and image signal for the position information of each target object. Therefore, the first target object and the second target object are determined in order of importance from high to low, and a first virtual sound and image signal and a second virtual sound and image signal are generated for the first target object and the second target object respectively. The generation method is consistent with the generation method of the virtual sound and image signal of the present embodiment. Therefore, when the first target object and the second target object move at the same time, the first virtual sound and image signal corresponding to the first target object can be output through the first speaker, and the second virtual sound and image signal corresponding to the second target object can be output through the second speaker, so that the sound heard by the user is richer and more three-dimensional, and synchronized with the moving positions of the multiple target objects in the picture.

[0089] As an example, Figure 6 The figure shows a schematic diagram of display content of a display device provided by an embodiment of the present invention. It includes a two-channel TV system, the trajectory of the airplane currently playing in the picture flies from the lower left corner to the upper right corner of the screen, three positions are selected, the audience sits 3 meters in front of the TV, and the screen is divided into multiple grid areas by parallel and vertical lines. The line connecting the center point of each grid area and the center point of the human head will form a different angle with the line connecting the center point of the screen and the center point of the human head. Each angle corresponds to a set of filters in the HRTF database. When the airplane in the picture is located at different positions ①②③, the SoC will automatically call the filter in the HRTF database corresponding to the corresponding grid area. After the signal is processed by the filter, the virtual sound image and the physical coordinates of ①②③ will coincide, so that the audience feels that the sound image position moves along the trajectory of the airplane, realizing sound and picture tracking.

[0090] As an example, Figure 7 FIG. 2 is a flow chart of another method for processing a sound source signal provided by an embodiment of the present invention. Figure 8The figure is a flow chart of another sound source signal processing method provided by an embodiment of the present invention, which obtains the grid area where the moving object in the TV screen is currently located, determines the filter in the HRTF database corresponding to the current grid area, inputs the signal into the HRTF filter, and processes the signal after the superposition filter through the crosstalk elimination algorithm, so that the position of the virtual sound and image signal output by the speaker coincides with the position of the moving object in the picture.

[0091] The sound source signal processing method provided by the embodiment of the present invention calculates the head-related transfer function from each grid area in the display device to the viewing position corresponding to the display device as the filter of each grid area; the viewing position and the filter of each grid area are stored in the database. The target movement trajectory of the target object in the display device is obtained; multiple grid areas of the display device are obtained; a preset number of target grid areas are determined from the grid areas through which the target movement trajectory passes; each target grid area is determined as a position information, and multiple position information are obtained. The filter corresponding to each position information is obtained, and each position information corresponds to a filter. The real-time position information of the target object during movement is obtained; the position information closest to the real-time position information is determined as the target position information; the sound source signal is superimposed with the filter corresponding to the target position information to obtain a new sound source signal; the new sound source signal is subjected to crosstalk elimination processing to obtain a virtual sound and image signal. Therefore, the audio and video tracking of dual-channel TVs can be achieved by introducing head-related transfer functions and crosstalk elimination technology. By dividing the TV screen into grids, HRTF data corresponding to the virtual sound and image that can be perceived at this position is stored in each grid area to provide filter parameters for the left and right channels. When the moving object is in different grid areas, the HRTF data of the corresponding area is called. When the HRTF data is switched, the listener feels that the sound is moving on the screen. The corresponding movement trajectory makes the virtual sound and image position coincide with the moving object on the screen, thereby improving the user experience.

[0092] Figure 3 A schematic diagram of the structure of a sound source signal processing device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the device specifically includes:

[0093] The determination module 31, when there is a moving target object in the display device, determines a plurality of position information of the target object in the display device;

[0094] An acquisition module 32, used for acquiring a filter corresponding to each piece of the position information, wherein each piece of the position information corresponds to a filter;

[0095] The processing module 33 is used to superimpose the sound source signal with the filter corresponding to each of the position information during the movement of the target object, so as to output the superimposed virtual sound and image signal through the speaker of the display device.

[0096] In a possible implementation manner, the determining module is specifically configured to obtain a target movement trajectory of the target object in the display device;

[0097] Acquire multiple grid areas of the display device;

[0098] Determine a preset number of target grid areas from the grid areas through which the target moving trajectory passes;

[0099] Each target grid area is determined as a piece of position information, and a plurality of pieces of position information are obtained.

[0100] In a possible implementation manner, the processing module is specifically used to obtain real-time position information of the target object during movement;

[0101] Determine the location information closest to the real-time location information as the target location information;

[0102] Superimposing the sound source signal with the filter corresponding to the target position information to obtain a new sound source signal;

[0103] The new sound source signal is subjected to crosstalk elimination processing to obtain the virtual sound and image signal.

[0104] In a possible implementation manner, the processing module is specifically used to obtain a system transfer function frequency response matrix from each of the speakers to a left ear and a right ear of a human body at a viewing position corresponding to the display device;

[0105] Performing an inverse operation on the frequency response matrix of the system transfer function to obtain a frequency response matrix of the crosstalk cancellation filter;

[0106] The new audio source signal is subjected to crosstalk cancellation processing according to the frequency response matrix of the crosstalk cancellation filter.

[0107] In a possible implementation manner, the processing module is further used to calculate a head-related transfer function from each grid area in the display device to a viewing position corresponding to the display device as a filter for each grid area;

[0108] storing the viewing position and the filter of each grid area in a database;

[0109] The acquisition module is specifically used to acquire the filter corresponding to each piece of the position information from the database.

[0110] In a possible implementation manner, the processing module is further configured to calculate a head-related transfer function from each grid area to a new viewing position as a new filter when the viewing position changes;

[0111] The database is updated according to the new filter and the new viewing position.

[0112] In a possible implementation manner, the determination module is further configured to determine the first target object and the second target object from the multiple target objects according to the importance of each target object when there are multiple target objects in the display device;

[0113] Determine first position information of the first target object and second position information of the second target object;

[0114] The processing module is further configured to superimpose a first sound source signal corresponding to the first target object and a filter corresponding to the first position information during the movement of the first target object, so as to output a superimposed first virtual sound and image signal through a first speaker of the display device;

[0115] During the movement of the second target object, the second sound source signal corresponding to the second target object is superimposed on the filter corresponding to the second position information, so as to output the superimposed second virtual sound and image signal through the second speaker of the display device.

[0116] The sound source signal processing device provided in this embodiment can be as follows Figure 3 The device shown in FIG. 1 can perform the following steps: Figure 1-2 All steps of the sound source signal processing method are realized Figure 1-2 For details, please refer to Figure 1-2 For the sake of brevity, the relevant description is not repeated here.

[0117] Figure 4 A schematic diagram of the structure of a display device provided by an embodiment of the present invention, Figure 4 The display device 400 shown includes: at least one processor 401, a memory 402, at least one network interface 404 and other user interfaces 403. The various components in the display device 400 are coupled together via a bus system 405. It is understood that the bus system 405 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 405 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 405 is not described in detail. Figure 4 Various buses are labeled as bus system 405 .

[0118] The user interface 403 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0119] It can be understood that the memory 402 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 402 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0120] In some implementations, the memory 402 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 4021 and application programs 4022 .

[0121] The operating system 4021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 4022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 4022.

[0122] In the embodiment of the present invention, by calling the program or instruction stored in the memory 402, specifically, the program or instruction stored in the application 4022, the processor 401 is used to execute the method steps provided by each method embodiment, for example, including:

[0123] When there is a moving target object in the display device, determining a plurality of position information of the target object in the display device;

[0124] Obtain a filter corresponding to each piece of the position information, wherein each piece of the position information corresponds to a filter;

[0125] During the movement of the target object, the sound source signal is superimposed on the filter corresponding to each of the position information, so as to output the superimposed virtual sound and image signal through the speaker of the display device.

[0126] In a possible implementation, obtaining a target movement trajectory of the target object in the display device;

[0127] Acquire multiple grid areas of the display device;

[0128] Determine a preset number of target grid areas from the grid areas through which the target moving trajectory passes;

[0129] Each target grid area is determined as a piece of position information, and a plurality of pieces of position information are obtained.

[0130] In a possible implementation, real-time position information of the target object during movement is obtained;

[0131] Determine the location information closest to the real-time location information as the target location information;

[0132] Superimposing the sound source signal with the filter corresponding to the target position information to obtain a new sound source signal;

[0133] The new sound source signal is subjected to crosstalk elimination processing to obtain the virtual sound and image signal.

[0134] In a possible implementation, a system transfer function frequency response matrix from each of the loudspeakers to a left ear and a right ear of a human body at a viewing position corresponding to the display device is obtained;

[0135] Performing an inverse operation on the frequency response matrix of the system transfer function to obtain a frequency response matrix of the crosstalk cancellation filter;

[0136] The new audio source signal is subjected to crosstalk cancellation processing according to the frequency response matrix of the crosstalk cancellation filter.

[0137] In a possible implementation, a head-related transfer function from each grid area in the display device to a viewing position corresponding to the display device is calculated as a filter for each grid area;

[0138] storing the viewing position and the filter of each grid area in a database;

[0139] The obtaining of the filter corresponding to each piece of the position information includes:

[0140] A filter corresponding to each piece of the position information is obtained from the database.

[0141] In a possible implementation, when the viewing position changes, a head-related transfer function from each grid area to the new viewing position is calculated as a new filter;

[0142] The database is updated according to the new filter and the new viewing position.

[0143] In a possible implementation, when there are multiple target objects in the display device, the first target object and the second target object are determined from the multiple target objects according to the importance of each target object;

[0144] Determine first position information of the first target object and second position information of the second target object;

[0145] During the movement of the first target object, superimposing a first sound source signal corresponding to the first target object and a filter corresponding to the first position information, so as to output a superimposed first virtual sound and image signal through a first speaker of the display device;

[0146] During the movement of the second target object, the second sound source signal corresponding to the second target object is superimposed on the filter corresponding to the second position information, so as to output the superimposed second virtual sound and image signal through the second speaker of the display device.

[0147] The method disclosed in the above embodiment of the present invention can be applied to the processor 401, or implemented by the processor 401. The processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 401 or the instruction in the form of software. The above processor 401 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 402, and the processor 401 reads the information in the memory 402 and completes the steps of the above method in combination with its hardware.

[0148] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0149] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0150] The display device provided in this embodiment may be Figure 4 The display device shown in FIG. 1 may execute the following Figure 1-2 All steps of the sound source signal processing method are realized Figure 1-2 For details, please refer to Figure 1-2 For the sake of brevity, the relevant description is not repeated here.

[0151] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0152] When one or more programs in the storage medium can be executed by one or more processors, the audio source signal processing method executed on the device side can be implemented.

[0153] The processor is used to execute the sound source signal processing program stored in the memory to implement the following steps of the sound source signal processing method executed on the device side:

[0154] When there is a moving target object in the display device, determining a plurality of position information of the target object in the display device;

[0155] Obtain a filter corresponding to each piece of the position information, wherein each piece of the position information corresponds to a filter;

[0156] During the movement of the target object, the sound source signal is superimposed on the filter corresponding to each of the position information, so as to output the superimposed virtual sound and image signal through the speaker of the display device.

[0157] In a possible implementation, obtaining a target movement trajectory of the target object in the display device;

[0158] Acquire multiple grid areas of the display device;

[0159] Determine a preset number of target grid areas from the grid areas through which the target moving trajectory passes;

[0160] Each target grid area is determined as a piece of position information, and a plurality of pieces of position information are obtained.

[0161] In a possible implementation, real-time position information of the target object during movement is obtained;

[0162] Determine the location information closest to the real-time location information as the target location information;

[0163] Superimposing the sound source signal with the filter corresponding to the target position information to obtain a new sound source signal;

[0164] The new sound source signal is subjected to crosstalk elimination processing to obtain the virtual sound and image signal.

[0165] In a possible implementation, a system transfer function frequency response matrix from each of the loudspeakers to a left ear and a right ear of a human body at a viewing position corresponding to the display device is obtained;

[0166] Performing an inverse operation on the frequency response matrix of the system transfer function to obtain a frequency response matrix of the crosstalk cancellation filter;

[0167] The new audio source signal is subjected to crosstalk cancellation processing according to the frequency response matrix of the crosstalk cancellation filter.

[0168] In a possible implementation, a head-related transfer function from each grid area in the display device to a viewing position corresponding to the display device is calculated as a filter for each grid area;

[0169] storing the viewing position and the filter of each grid area in a database;

[0170] The obtaining of the filter corresponding to each piece of the position information includes:

[0171] A filter corresponding to each piece of the position information is obtained from the database.

[0172] In a possible implementation, when the viewing position changes, a head-related transfer function from each grid area to the new viewing position is calculated as a new filter;

[0173] The database is updated according to the new filter and the new viewing position.

[0174] In a possible implementation, when there are multiple target objects in the display device, the first target object and the second target object are determined from the multiple target objects according to the importance of each target object;

[0175] Determine first position information of the first target object and second position information of the second target object;

[0176] During the movement of the first target object, superimposing a first sound source signal corresponding to the first target object and a filter corresponding to the first position information, so as to output a superimposed first virtual sound and image signal through a first speaker of the display device;

[0177] During the movement of the second target object, the second sound source signal corresponding to the second target object is superimposed on the filter corresponding to the second position information, so as to output the superimposed second virtual sound and image signal through the second speaker of the display device.

[0178] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0179] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0180] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing a sound source signal, It is characterized in that include: When there is a moving target object in the display device, determining a plurality of position information of the target object in the display device; Obtain a filter corresponding to each piece of the position information, wherein each piece of the position information corresponds to a filter; During the movement of the target object, the sound source signal is superimposed with the filter corresponding to each of the position information, so as to output the superimposed virtual sound and image signal through the speaker of the display device.

2. The method according to claim 1, It is characterized in that The determining a plurality of position information of the target object in the display device includes: Acquire a target movement trajectory of the target object in the display device; Acquire multiple grid areas of the display device; Determine a preset number of target grid areas from the grid areas through which the target moving trajectory passes; Each target grid area is determined as a piece of position information, and a plurality of pieces of position information are obtained.

3. The method according to claim 1, It is characterized in that During the movement of the target object, superimposing the sound source signal with the filter corresponding to each of the position information includes: Acquiring real-time location information of the target object during its movement; Determine the location information closest to the real-time location information as the target location information; Superimposing the sound source signal with the filter corresponding to the target position information to obtain a new sound source signal; The new sound source signal is subjected to crosstalk elimination processing to obtain the virtual sound and image signal.

4. The method according to claim 3, It is characterized in that The performing crosstalk elimination processing on the new audio source signal comprises: Obtaining a system transfer function frequency response matrix from each of the loudspeakers to the left ear and the right ear of a human body at a viewing position corresponding to the display device; Performing an inverse operation on the frequency response matrix of the system transfer function to obtain a frequency response matrix of the crosstalk cancellation filter; The new audio source signal is subjected to crosstalk cancellation processing according to the frequency response matrix of the crosstalk cancellation filter.

5. The method according to claim 2, It is characterized in that Before determining a plurality of position information of the target object in the display device, the method further includes: Calculating a head-related transfer function from each grid area in the display device to a viewing position corresponding to the display device as a filter for each grid area; storing the viewing position and the filter of each grid area in a database; The obtaining of the filter corresponding to each piece of the position information includes: A filter corresponding to each piece of the position information is obtained from the database.

6. The method according to claim 5, It is characterized in that The method further comprises: When the viewing position changes, a head-related transfer function from each grid area to the new viewing position is calculated as a new filter; The database is updated according to the new filter and the new viewing position.

7. The method according to claim 1, It is characterized in that The method further comprises: When there are multiple target objects in the display device, determining a first target object and a second target object from the multiple target objects according to the importance of each target object; Determine first position information of the first target object and second position information of the second target object; During the movement of the first target object, superimposing a first sound source signal corresponding to the first target object and a filter corresponding to the first position information, so as to output a superimposed first virtual sound and image signal through a first speaker of the display device; During the movement of the second target object, the second sound source signal corresponding to the second target object is superimposed on the filter corresponding to the second position information, so as to output the superimposed second virtual sound and image signal through the second speaker of the display device.

8. A sound source signal processing device, It is characterized in that include: When there is a moving target object in the display device, determining a plurality of position information of the target object in the display device; Obtain a filter corresponding to each piece of the position information, wherein each piece of the position information corresponds to a filter; During the movement of the target object, the sound source signal is superimposed with the filter corresponding to each of the position information, so as to output the superimposed virtual sound and image signal through the speaker of the display device.

9. A display device, It is characterized in that include: A processor and a memory, wherein the processor is used to execute a sound source signal processing program stored in the memory to implement the sound source signal processing method according to any one of claims 1 to 7.

10. A storage medium, It is characterized in that The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the sound source signal processing method according to any one of claims 1 to 7.

Images