A sound effect matching method, device, multimedia playback device and storage medium

Abstract

The embodiment of the present application discloses a sound effect matching method, which includes: obtaining spatial information of a playback scene in which a multimedia playback device is located; determining a sound field sound effect that matches the spatial information; obtaining a first sound source sound effect that matches a first audio information in the multimedia information to be played by the multimedia playback device; based on the sound field sound effect and the first sound source sound effect, determining a first output sound effect of the first audio information, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played. The embodiment of the present application also discloses a sound effect matching device, a multimedia playback device, and a storage medium.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of computer technology, and in particular to a sound effect matching method, a sound effect matching device, a multimedia playback device and a storage medium. Background Art

[0002] Multimedia playback devices, such as smart TVs, have various pre-made sound effects, such as standard, cinema, music, custom, and some even have subdivided modes such as classical and rock. Currently, users use a remote control to operate the sound effect mode option in the user interface (UI) menu of the multimedia playback device, select the desired sound effect mode, and click confirm to make it effective. In other words, currently multimedia playback devices only support users to manually select the output sound effects of multimedia information. Summary of the invention

[0003] The embodiments of the present application hope to provide a sound effect matching method, a sound effect matching device, a multimedia playback device and a storage medium, which solve the problem in the related art that the multimedia playback device only supports the user to manually select the output sound effect of the multimedia information, and realizes the combination of the sound field sound effect corresponding to the spatial information of the playback scene in which the multimedia playback device is located and the sound source sound effect matched with the audio information in the multimedia information to be played, and jointly determines the optimal output sound effect of the audio information, so as to realize automatic matching and confirmation of the optimal playback sound effect of the multimedia information, facilitate the user's operation, and enhance the user experience.

[0004] The technical solution of this application is implemented as follows:

[0005] A sound effect matching method, the method comprising:

[0006] Obtaining spatial information of a playback scene where a multimedia playback device is located;

[0007] Determining a sound field sound effect that matches the spatial information;

[0008] Acquire a first sound source sound effect that matches the first audio information in the multimedia information to be played by the multimedia playback device;

[0009] Based on the sound field sound effect and the first sound source sound effect, a first output sound effect of the first audio information is determined, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played.

[0010] A sound effect matching device, the sound effect matching device comprising:

[0011] An acquisition module, used to acquire spatial information of a playback scene where a multimedia playback device is located;

[0012] A processing module, used to determine a sound field sound effect matching the spatial information;

[0013] The acquisition module is further used to acquire a first sound source sound effect that matches the first audio information in the multimedia information to be played by the multimedia playback device;

[0014] The processing module is further used to determine a first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played.

[0015] A multimedia playback device, comprising: a processor, a memory and a communication bus;

[0016] The communication bus is used to realize the communication connection between the processor and the memory;

[0017] The processor is used to execute the sound effect matching program stored in the memory to implement the steps of the above-mentioned sound effect matching method.

[0018] A storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the above-mentioned sound effect matching method.

[0019] The embodiments of the present application provide a sound effect matching method, a sound effect matching device, a multimedia playback device and a storage medium, wherein the method comprises: obtaining spatial information of a playback scene in which the multimedia playback device is located; determining a sound field sound effect that matches the spatial information; obtaining a first sound source sound effect that matches the first audio information in the multimedia information to be played by the multimedia playback device; determining a first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played; combining the sound field sound effect corresponding to the spatial information of the playback scene in which the multimedia playback device is located and the sound source sound effect that matches the audio information in the multimedia information to be played, jointly determining the best output sound effect of the audio information, thereby automatically matching and confirming the best playback sound effect of the multimedia information, facilitating user operation and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the process of the sound effect matching method provided in the embodiment of the present application Figure 1 ;

[0021] Figure 2 Schematic diagram of the process of the sound effect matching method provided in the embodiment of the present application Figure 2 ;

[0022] Figure 3 Schematic diagram of the process of the sound effect matching method provided in the embodiment of the present application Figure 3;

[0023] Figure 4 Schematic diagram of the process of the sound effect matching method provided in the embodiment of the present application Figure 4 ;

[0024] Figure 5 Schematic diagram of the process of the sound effect matching method provided in the embodiment of the present application Figure 5 ;

[0025] Figure 6 Schematic diagram of the process of the sound effect matching method provided in the embodiment of the present application Figure 6 ;

[0026] Figure 7 Schematic diagram of the process of the sound effect matching method provided in the embodiment of the present application Figure 7 ;

[0027] Figure 8 A schematic diagram of the structure of a sound effect matching device provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the structure of a multimedia playback device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0030] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0033] The embodiment of the present application provides a sound effect matching method, which is applied to a multimedia playback device, referring to Figure 1 As shown, the method comprises the following steps:

[0034] Step 101: Acquire spatial information of a playback scene where a multimedia playback device is located.

[0035] In an embodiment of the present application, the multimedia playback device may include mobile terminal devices such as laptop computers, personal digital assistants (PDA), car audio and video equipment, and fixed terminal devices such as smart TVs, conference equipment, desktop computers, projection equipment, etc.

[0036] Here, the playback scene of the multimedia playback device includes closed playback scenes such as conference rooms, bedrooms, car compartments, home theaters, etc., and open playback scenes such as shopping malls, waiting halls, stadiums and other open-air venues. Spatial information includes but is not limited to the length and width of the playback scene; the structure points and structure lines of the playback scene, which are the connection points and connection lines of different planes in the playback scene.

[0037] Step 102: Determine the sound field sound effect that matches the spatial information.

[0038] In the embodiment of the present application, the sound field refers to the area where sound waves exist in the medium. In the present application, the place where the scene is played is the sound field.

[0039] In the embodiments of the present application, sound effects refer to the effects produced by sound. Sound field sound effects refer to the sound effects matched with the spatial information of the playback scene. In other words, when determining the output sound effects of the multimedia information to be played, the present application refers to the actual scene in which the multimedia playback device is located.

[0040] Step 103: Acquire a first sound source sound effect that matches the first audio information in the multimedia information to be played by the multimedia playback device.

[0041] In the embodiment of the present application, the sound source sound effect is a sound effect determined by the multimedia information to be played itself. The sound source sound effect can be determined according to the source format information of the multimedia information to be played. The sound source sound effect can also be pre-set according to the expression content of the multimedia information to be played.

[0042] Step 104: determine a first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played.

[0043] In the embodiment of the present application, when determining the output sound effect of the multimedia information to be played, not only the actual scene in which the multimedia playback device is located is taken into consideration, and the sound field sound effect is matched based on the spatial information of the actual scene; but also the characteristics of the multimedia information to be played itself are taken into consideration, and the sound source sound effect is matched based on the characteristics of the multimedia information to be played itself, and finally based on the characteristics of the two different dimensions of sound field sound effect and sound source sound effect, the first output sound effect of the multimedia information to be played is comprehensively determined, and the purpose of automatically matching the best sound effect is achieved. It should be noted that the execution order of the steps of determining the sound field sound effect and obtaining the first sound source sound effect in the embodiment of the present application is not specifically limited by this application, and is subject to the implementation of the sound effect matching method provided in this application.

[0044] The sound effect matching method provided in the embodiment of the present application includes: obtaining spatial information of a playback scene in which a multimedia playback device is located; determining a sound field sound effect that matches the spatial information; obtaining a first sound source sound effect that matches the first audio information in the multimedia information to be played by the multimedia playback device; based on the sound field sound effect and the first sound source sound effect, determining a first output sound effect of the first audio information, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played; solving the problem in the related art that the multimedia playback device only supports the user to manually select the output sound effect of the multimedia information, and realizing the combination of the sound field sound effect corresponding to the spatial information of the playback scene in which the multimedia playback device is located and the sound source sound effect that matches the audio information in the multimedia information to be played, and jointly determining the best output sound effect of the audio information, so that the best playback sound effect of the multimedia information can be automatically matched and confirmed, which facilitates the user's operation and improves the user experience.

[0045] In other embodiments of the present application, step 102 of determining the sound field sound effect matching the spatial information can be implemented by the following steps:

[0046] The first step is to build a spatial sound field model of the playback scene based on spatial information.

[0047] In the embodiment of the present application, the spatial information is used to construct a spatial generation model of the playback scene. The spatial sound field model can be regarded as a three-dimensional spatial model of the playback scene. After obtaining the spatial information of the playback scene, the multimedia playback device can construct the spatial sound field model of the playback scene based on the spatial information.

[0048] The second step is to determine the sound field effects that match the spatial sound field model.

[0049] In an embodiment of the present application, after acquiring the spatial information, the multimedia playback device constructs a spatial sound field model of the playback scene based on the spatial information, which can more accurately restore the playback scene in which the multimedia playback device is located, and then match a more appropriate sound field sound effect based on the restored playback scene.

[0050] The embodiment of the present application provides a sound effect matching method, which is applied to a multimedia playback device, referring to Figure 2 As shown, the method comprises the following steps:

[0051] Step 201: Acquire spatial information of a playback scene where a multimedia playback device is located.

[0052] Step 202: Acquire the sound field environment information of the playback scene where the multimedia playback device is located.

[0053] In the embodiment of the present application, while drawing on the actual scene in which the multimedia playback device is located, the sound field environment information in the actual scene is also introduced, and the sound field environment information refers to the sound environment of the playback scene, such as a noisy environment, a quiet environment, a closed environment, an open environment, etc. Thus, the accuracy of the subsequent construction of the spatial sound field model is improved.

[0054] Step 203: construct a spatial sound field model of the playback scene based on the sound field environment information and the spatial information.

[0055] Step 204: Determine the sound field sound effect that matches the spatial sound field model.

[0056] Step 205: Obtain the format information of the first source of the multimedia information to be played.

[0057] In the embodiment of the present application, exemplarily, the first source format information of the multimedia information to be played includes but is not limited to the following formats: Dolby Atmos, DTS-HD Master Audio, TRUE HD, Moving Picture Experts Group 4 (Mp4), Advanced Audio Coding (aac), ddp, Moving Picture Experts Group Audio Layer III (Mp3), and waveform sound file (wav).

[0058] Step 206: Determine a first sound source sound effect that matches the first audio information based on the first source format information.

[0059] In the embodiment of the present application, when determining the first sound source sound effect that matches the first audio information, it is determined according to the source format information of the multimedia information to be played. Based on the above-mentioned first source format information as an example, the first sound source sound effect that matches the first audio information determined based on the first source format information is further described here; the first sound source sound effect determined by the first source format information of the multimedia playback device based on the audio (AMP) of the multimedia playback device for the Dolby Atmos format is the cinema virtual surround plus Atmos sound effect; the first sound source sound effect determined by the first source format information of the DTS-HD Master Audio format is the cinema virtual surround plus DTS sound effect; the first sound source sound effect determined by the first source format information of the TRUE HD format is the cinema virtual surround plus THD sound effect; the first sound source sound effect determined by the first source format information of the Mp4, aac, ddp format is the cinema virtual surround sound effect; the first sound source sound effect determined by the first source format information of the pure digital sound (audio) format such as Mp3, wav is the music mode sound effect.

[0060] Step 207: superimpose the sound field sound effect and the first sound source sound effect based on a sound effect synthesis algorithm to obtain a first output sound effect.

[0061] In the embodiment of the present application, the multimedia playback device, after determining the sound field sound effect and the first sound source sound effect, superimposes the sound field sound effect and the first sound source sound effect, and finally outputs an optimal synthesized sound effect.

[0062] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0063] The embodiment of the present application provides a sound effect matching method, which is applied to a multimedia playback device, referring to Figure 3 As shown, the method comprises the following steps:

[0064] Step 301: After the sound wave transmitting component in the playback scene transmits a detection sound wave signal, an echo signal received by the sound wave receiving component is obtained.

[0065] Among them, the echo signal is a signal obtained by reflecting from the surface of an object when the detection sound wave signal encounters an object during the propagation process.

[0066] In the embodiment of the present application, when an ultrasonic imaging system is provided in the multimedia playback device, the detection sound wave signal is an ultrasonic signal; when a microwave imaging system is provided in the multimedia playback device, the detection sound wave signal is a microwave signal. In the embodiment of the present application, the type of the sound wave imaging system is not specifically limited.

[0067] Step 302: Generate a scene image of the playback scene based on the echo signal.

[0068] In an embodiment of the present application, when the detection sound wave signal encounters an object during propagation, it will be reflected on the surface of the object, thereby returning a corresponding echo signal. The multimedia playback device receives the echo signal through a built-in sound wave receiving component and uses a sound wave imaging algorithm to draw a scene image of the object, i.e., the playback scene.

[0069] Step 303: Input the scene image into the trained neural network model to obtain the structural points and structural lines of the playback scene.

[0070] Among them, the structure points and structure lines are the connection points and connection lines of different planes in the playback scene.

[0071] In the embodiment of the present application, the neural network model is trained based on the correspondence between the panoramic image of a previously known room and the corresponding three-dimensional image. Alternatively, the neural network model is trained based on the correspondence between the panoramic image acquired in advance and the corresponding structural points and structural lines in the room.

[0072] Here, structural points and structural lines are the connection points and connection lines of different planes in the room. For example, structural lines include the boundary lines between wall and ceiling, wall and floor, and the boundary between walls. Structural points include the intersection points between adjacent walls and ceiling or adjacent walls and floor.

[0073] In the embodiment of the present application, the neural network model is trained in advance using the existing massive data. The model inputs the panoramic image of the playback scene and outputs the structural lines related to the three-dimensional structure in the panoramic image, such as the intersection lines between the wall and the ceiling, the wall and the floor, etc., and the intersection positions of adjacent walls and the ceiling or floor.

[0074] Step 304: Match the scene images based on the structure points and structure lines to obtain the position and posture relationship of the scene images at all points.

[0075] Here, multiple panoramic images acquired from a room can be spliced ​​according to the structural lines and structural points to obtain the position and posture relationships of all the panoramic image points corresponding to the room. The panoramic images can be spliced ​​according to these position and posture relationships.

[0076] Step 305: Based on the posture relationship, structure points and structure lines, a three-dimensional image of the playback scene is formed by splicing.

[0077] The spatial information includes three-dimensional images.

[0078] Here, according to the position relationship of multiple panoramic images, the panoramic images are stitched to obtain a panoramic image of the playback scene, such as the entire room. Combined with the relative positions of the structure points and structure lines in the panoramic image, the panoramic image is processed to obtain a three-dimensional image of the playback scene.

[0079] Step 306: construct a spatial sound field model of the playback scene based on the spatial information.

[0080] Step 307: Determine the sound field sound effect that matches the spatial sound field model.

[0081] Step 308: Acquire a first sound source sound effect that matches the first audio information in the multimedia information to be played by the multimedia playback device.

[0082] Step 309: determine a first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played.

[0083] In a feasible scenario, taking the multimedia playback device as a smart TV as an example, combined with Figure 4 As shown, in step 10, the built-in sound wave detection system is turned on after the smart TV is turned on. Step 11. The smart TV emits sound waves to detect the spatial information of the current playback scene of the smart TV, such as the spatial structure, and obtains the sound field environment information. Step 12. The smart TV automatically builds the current spatial sound field model based on the sound field environment information and spatial information. In actual applications, the spatial sound field model includes but is not limited to empty conference rooms, enclosed listening rooms, noisy spaces, and other models, and the sound field conditions of different spaces vary greatly. Step 13. The smart TV matches the pre-made sound field sound effect A in the smart TV according to the current spatial sound field model automatically established. Step 14. The user selects an audio or video source to play, and the smart TV determines the selected audio or video source. Step 15. The smart TV parses the source format of the audio and video source. For different formats, such as step 16, Dolby Atmos format is matched to sound source sound effect B1, DTS-HD format is matched to sound source sound effect B2, TRUE HD format is matched to sound source sound effect B3, Mp4, aac and dpp formats are matched to sound source sound effect B3, pure audio such as Mp3, wav format is matched to sound source sound effect B4. Step 17, the smart TV superimposes the aforementioned sound field sound effect A and sound source sound effect Bn, and finally outputs an optimal synthesized sound effect, and controls the playback module to output the audio and video source with the optimal synthesized sound effect. Among them, Bn includes the above B1-B4. In particular, the sound field sound effect A is established and matched according to the environment in which the smart TV is located. As long as the smart TV does not move, it generally does not change, while the sound source sound effect Bn changes with different sources.

[0084] It can be seen that the sound effect matching method provided by this application does not require users to select various sound effect settings on the UI of the smart TV. All are automatically identified and processed by the smart TV system. The sound field and sound effect mode can be intelligently matched according to the sound field conditions of the user's environment and the type of audio and video source, allowing users to experience the best sound effect.

[0085] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0086] The embodiment of the present application provides a sound effect matching method, which is applied to a multimedia playback device, referring to Figure 5 As shown, the method comprises the following steps:

[0087] Step 401: Obtain a panoramic image captured by an image acquisition module in a playback scene.

[0088] The panoramic image includes at least two two-dimensional images taken from different shooting angles.

[0089] In the embodiment of the present application, the panoramic image includes a two-dimensional image captured by at least two image acquisition modules in the playback scene, and the shooting angles of the at least two image acquisition modules are different; here, the shooting angles of the at least two image acquisition modules are different, which can be understood as that the shooting angles of the at least two image acquisition modules do not completely overlap, or the shooting angles of the at least two image acquisition modules do not overlap. Exemplarily, the shooting angles of the at least two image acquisition modules can cover all positions of the playback scene as much as possible.

[0090] Step 402: Calibrate the image acquisition module.

[0091] In an embodiment of the present application, the image acquisition module is calibrated. The image acquisition module can be used to photograph objects in the above-mentioned playback scene or other playback scenes, and then the image acquisition module is calibrated based on at least the photographed image containing the object and the actual size of the photographed object. The calibrated image acquisition module has internal parameters and external parameters.

[0092] Step 403: Obtain the internal parameters and external parameters of the calibrated image acquisition module.

[0093] Among them, the external parameters are used to indicate the conversion relationship between the coordinate system of the image acquisition module and the world coordinate system, and the internal parameters are used to indicate the conversion relationship between the coordinate system of the image acquisition module and the image coordinate system.

[0094] Here, after calibrating the image acquisition module, the internal and external parameters of the image acquisition module can be obtained. Here, the origin of the coordinate system of the image acquisition module is located at the optical center of the imaging module, the vertical axis, i.e., the Z axis, coincides with the optical axis of the imaging module, and the horizontal axis, i.e., the X axis, and the longitudinal axis, i.e., the Y axis, are parallel to the imaging plane. The world coordinate system includes the latitude and longitude coordinate system.

[0095] Step 404: Determine the length, width and height of the playback scene based on the internal parameters, external parameters and the two-dimensional image captured by the image acquisition module.

[0096] The spatial information includes length, width and height.

[0097] Step 405: construct a spatial sound field model of the playback scene based on the spatial information.

[0098] Step 406: Determine the sound field sound effect that matches the spatial sound field model.

[0099] Step 407: Acquire a first sound source sound effect that matches the first audio information in the multimedia information to be played by the multimedia playback device.

[0100] Step 408: Determine a first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played.

[0101] In other embodiments of the present application, after step 309 determines the first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played, or step 408 determines the first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played, the following can also be performed: Figure 6 Step 501-step 502, or step 501 and step 503-step 505 shown:

[0102] Step 501: Detect location change information of a multimedia playback device.

[0103] Step 502: If the position change information indicates that the multimedia playback device has not moved, the multimedia information to be played having the first source format information is output with a first output sound effect.

[0104] Step 503: If the position change information indicates that the multimedia device has moved, obtain spatial information of the multimedia device after the movement.

[0105] Step 504: Determine the sound field effect after the movement that matches the spatial information after the movement.

[0106] Step 505: based on the moved sound field sound effect and the first sound source sound effect, update the first output sound effect of the first audio information, so that the first audio information is output with the updated first output sound effect when the multimedia information to be played is played.

[0107] In other embodiments of the present application, after step 309 determines the first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played, or step 408 determines the first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played, the following can also be performed: Figure 7 Steps 601-603, or steps 601-602 and steps 604-605 shown:

[0108] Step 601: Obtain a switching event for multimedia information to be played.

[0109] Step 602: In response to the switching event, obtain the format information of the second source of the multimedia information to be played after the switching.

[0110] Step 603: If the second source format information is the same as the first source format information, determine that the output sound effect of the switched multimedia information to be played is the first output sound effect, so that when playing the switched multimedia information to be played, the second audio information in the switched multimedia information to be played is output with the first output sound effect.

[0111] Step 604: If the second source format information is different from the first source format information, determine a second sound source sound effect that matches the second audio information based on the second source format information.

[0112] Step 605: Determine a second output sound effect of the second audio information based on the sound field sound effect and the second sound source sound effect, so that when playing the switched multimedia information to be played, the second audio information in the switched multimedia information to be played is output with the second output sound effect.

[0113] It can be seen that in this embodiment, the output sound effects of the multimedia information to be played corresponding to the same source format information in the same playback scenario can be quickly reused; for the multimedia information to be played corresponding to different source format information in the same playback scenario, at least the sound field sound effects can be quickly reused, thereby improving the efficiency and accuracy of the multimedia playback device in matching the output sound effects.

[0114] It should be noted that in actual applications, the position movement of the multimedia playback device and the switching of the multimedia information to be played can exist at the same time. At this time, the sound field sound effects and the sound source sound effects need to be redetermined based on the new spatial information of the multimedia playback device after the movement and the source format of the multimedia information to be played after the switching.

[0115] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0116] The embodiment of the present application provides a sound effect matching device, which can be applied to Figure 1-3 , 5 In a sound effect matching method provided in the corresponding embodiment, referring to Figure 8 As shown, the sound effect matching device 7 includes:

[0117] The acquisition module 701 is used to acquire the spatial information of the playback scene where the multimedia playback device is located;

[0118] A processing module 702 is used to determine a sound field sound effect that matches the spatial information;

[0119] The acquisition module 701 is further used to acquire a first sound source sound effect that matches the first audio information in the multimedia information to be played by the multimedia playback device;

[0120] The processing module 702 is further used to determine a first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played.

[0121] In other embodiments of the present application, the processing module 702 is further used to construct a spatial sound field model of the playback scene based on the spatial information; and determine the sound field sound effect that matches the spatial sound field model.

[0122] In other embodiments of the present application, the acquisition module 701 is also used to obtain the sound field environment information of the playback scene in which the multimedia playback device is located; the processing module 702 is also used to construct a spatial sound field model of the playback scene based on the sound field environment information and spatial information.

[0123] In other embodiments of the present application, the processing module 702 is further used to perform superposition processing on the sound field sound effect and the first sound source sound effect based on a sound effect synthesis algorithm to obtain a first output sound effect.

[0124] In other embodiments of the present application, the acquisition module 701 is further used to obtain the first source format information of the multimedia information to be played; the processing module 702 is further used to determine the first sound source sound effect matching the first audio information based on the first source format information.

[0125] In other embodiments of the present application, the acquisition module 701 is also used to obtain the echo signal received by the sound wave receiving component after the sound wave transmitting component in the playback scene transmits the detection sound wave signal, wherein the echo signal is a signal obtained by reflecting the surface of the object when the detection sound wave signal encounters an object during propagation; the processing module 702 is also used to generate a scene image of the playback scene based on the echo signal; the scene image is input into a trained neural network model to obtain the structural points and structural lines of the playback scene, wherein the structural points and structural lines are the connection points and connection lines of different planes in the playback scene; the scene image is matched based on the structural points and structural lines to obtain the posture relationship of the scene images of all points; based on the posture relationship, the structural points and structural lines, a three-dimensional image of the playback scene is spliced, wherein the spatial information includes the three-dimensional image.

[0126] In other embodiments of the present application, the acquisition module 701 is also used to acquire a panoramic image acquired by an image acquisition module in the playback scene, wherein the panoramic image includes at least two two-dimensional images taken from different shooting angles; the processing module 702 is also used to calibrate the image acquisition module; the acquisition module 701 is also used to acquire internal parameters and external parameters of the calibrated image acquisition module, wherein the external parameters are used to indicate the conversion relationship between the coordinate system of the image acquisition module and the world coordinate system, and the internal parameters are used to indicate the conversion relationship between the coordinate system of the image acquisition module and the image coordinate system; the processing module 702 is also used to determine the length of the playback scene, the width of the playback scene and the height of the playback scene based on the internal parameters, the external parameters and the two-dimensional image taken by the image acquisition module, wherein the spatial information includes the length, width and height.

[0127] In other embodiments of the present application, the sound effect matching device 7 also includes: a playback module 703, which is used to play the multimedia information to be played, and output the first audio information with a first output sound effect; an acquisition module 701, which is also used to obtain a switching event for the multimedia information to be played if the multimedia playback device has not moved; in response to the switching event, obtain the second source format information of the multimedia information to be played after the switching; the playback module 703 is also used to play the multimedia information to be played after the switching if the second source format information is the same as the first source format information, and output the second audio information in the switched multimedia information to be played with the first output sound effect.

[0128] In other embodiments of the present application, the processing module 702 is also used to determine, based on the second source format information, a second sound source sound effect that matches the second audio information if the second source format information is different from the first source format information; and to determine, based on the sound field sound effect and the second sound source sound effect, a second output sound effect of the second audio information; and the playing module 703 is also used to play the switched multimedia information to be played, and output the second audio information in the switched multimedia information to be played with the second output sound effect.

[0129] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0130] The embodiment of the present application provides a multimedia playback device, which can be applied to Figure 1-3 , 5 In a sound effect matching method provided in the corresponding embodiment, referring to Fig. 9 As shown, the multimedia playback device 8 ( Fig. 9 The multimedia player 8 in Figure 8 The sound effect matching device 7 in FIG. 1 includes: a processor 801, a memory 802 and a communication bus 803, wherein:

[0131] The communication bus 803 is used to implement the communication connection between the processor 801 and the memory 802 .

[0132] The processor 801 is used to execute the sound effect matching program stored in the memory 802 to achieve Figure 1-3 , 5 are steps in a sound effect matching method provided in the corresponding embodiment.

[0133] As an example, processor 801 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0134] It should be noted that the specific implementation process of the steps executed by the processor in this embodiment can refer to Figure 1-3 The implementation process of the sound effect matching method provided in the corresponding embodiment 5 will not be repeated here.

[0135] The embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the following. Figure 1-3 The implementation process of the sound effect matching method provided in the corresponding embodiment 5 will not be repeated here.

[0136] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0137] It should be understood that the "one embodiment" or "an embodiment" or "an embodiment of the present application" or "the aforementioned embodiment" or "some embodiments" or "some implementation methods" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "an embodiment of the present application" or "the aforementioned embodiment" or "some embodiments" or "some implementation methods" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0138] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0139] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0140] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0141] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0142] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0143] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0144] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0145] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0146] It is worth noting that the drawings in the embodiments of the present application are only for illustrating the schematic positions of various components on the terminal device and do not represent the actual positions in the terminal device. The actual positions of various components or areas may be changed or offset accordingly according to actual conditions (for example, the structure of the terminal device), and the proportions of different parts of the terminal device in the drawings do not represent the actual proportions.

[0147] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A sound effect matching method, characterized in that: The method comprises: Obtaining spatial information of a playback scene where a multimedia playback device is located; Determining a sound field sound effect that matches the spatial information; Acquire a first audio information matching the first audio information in the multimedia information to be played by the multimedia playback device First source sound effects; Determine a first input of the first audio information based on the sound field sound effect and the first sound source sound effect. Output sound effects, so that when playing the multimedia information to be played, the first audio information is output with the first output sound effect; The determining, based on the sound field sound effect and the first sound source sound effect, a first output sound effect of the first audio information comprises: Performing superposition processing on the sound field sound effect and the first sound source sound effect based on a sound effect synthesis algorithm to obtain the first output sound effect; The step of obtaining a first sound source sound effect that matches the first audio information in the multimedia information to be played by the multimedia playback device includes: Acquire the first source format information of the multimedia information to be played; Based on the first source format information, determine the first sound source sound effect that matches the first audio information.

2. The method according to claim 1, characterized in that The determining of the sound field sound effect matching the spatial information includes: Constructing a spatial sound field model of the playback scene based on the spatial information; The sound field sound effect that matches the spatial sound field model is determined.

3. The method according to claim 2, characterized in that The method further comprises: Acquiring the sound field environment information of the playback scene where the multimedia playback device is located; The step of constructing the spatial sound field model of the playback scene based on the spatial information includes: A spatial sound field model of the playback scene is constructed based on the sound field environment information and the spatial information.

4. The method according to any one of claims 1 to 3, characterized in that The obtaining of spatial information of a playback scene in which the multimedia playback device is located includes: After the sound wave transmitting component in the playback scene transmits the detection sound wave signal, an echo signal received by the sound wave receiving component is obtained, wherein the echo signal is a signal obtained by reflection from the surface of an object when the detection sound wave signal encounters an object during propagation; generating a scene image of the playback scene based on the echo signal; Inputting the scene image into a trained neural network model to obtain structure points and structure lines of the playback scene, wherein the structure points and the structure lines are connection points and connection lines of different planes in the playback scene; Matching the scene image based on the structure points and the structure lines to obtain the position and posture relationship of the scene image at all points; Based on the posture relationship, the structure points and the structure lines, a three-dimensional image of the playback scene is spliced ​​and formed, wherein the spatial information includes the three-dimensional image.

5. The method according to any one of claims 1 to 3, characterized in that The obtaining of spatial information of a playback scene in which the multimedia playback device is located includes: Acquire a panoramic image captured by an image acquisition module in the playback scene, wherein the panoramic image includes at least two two-dimensional images captured from different shooting angles; Calibrate the image acquisition module; Acquire the internal parameters and external parameters of the image acquisition module after calibration, wherein the external parameters are used to indicate the conversion relationship between the coordinate system of the image acquisition module and the world coordinate system, and the internal parameters are used to indicate the conversion relationship between the coordinate system of the image acquisition module and the image coordinate system; Based on the internal parameters, external parameters and the two-dimensional image captured by the image acquisition module, the length of the playback scene, the width of the playback scene and the height of the playback scene are determined, wherein the spatial information includes the length, the width and the height.

6. The method according to claim 1, characterized in that After determining the first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, the method further includes: Detecting position change information of the multimedia playback device; If the position change information indicates that the multimedia playback device has not moved, outputting the to-be-played multimedia information having the first source format information with the first output sound effect; If the position change information indicates that the multimedia playback device has moved, obtaining spatial information of the multimedia playback device after the movement; Determining a sound field sound effect after the movement that matches the spatial information after the movement; Based on the moved sound field sound effect and the first sound source sound effect, the first output sound effect of the first audio information is updated so that the first audio information is output with the updated first output sound effect when the multimedia information to be played is played.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining a switching event for the multimedia information to be played; In response to the switching event, obtaining format information of a second source of the multimedia information to be played after the switching; If the second source format information is the same as the first source format information, determining that the output sound effect of the switched multimedia information to be played is the first output sound effect, so that when the switched multimedia information to be played is played, the second audio information in the switched multimedia information to be played is output with the first output sound effect; If the second film source format information is different from the first film source format information, determining a second sound source sound effect matching the second audio information based on the second film source format information; Based on the sound field sound effect and the second sound source sound effect, a second output sound effect of the second audio information is determined so that when the switched multimedia information to be played is played, the second audio information in the switched multimedia information to be played is output with the second output sound effect.

8. A sound effect matching device, characterized in that: The sound effect matching device comprises: An acquisition module, used to acquire spatial information of a playback scene where a multimedia playback device is located; A processing module, used to determine a sound field sound effect matching the spatial information; The acquisition module is further used to acquire a first sound source sound effect that matches the first audio information in the multimedia information to be played by the multimedia playback device; The processing module is further used to determine a first output sound effect of the first audio information based on the sound field sound effect and the first sound source sound effect, so that the first audio information is output with the first output sound effect when the multimedia information to be played is played. The processing module is further used to perform superposition processing on the sound field sound effect and the first sound source sound effect based on a sound effect synthesis algorithm to obtain the first output sound effect; The acquisition module is further used to acquire the first source format information of the multimedia information to be played; The processing module is further used to determine the first sound source sound effect matching the first audio information based on the first film source format information.

9. A multimedia playback device, characterized in that: The multimedia playback device includes: a processor, a memory and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the sound effect matching program stored in the memory to implement the steps of the sound effect matching method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the sound effect matching method as described in any one of claims 1 to 7.

Images