A method and system for determining sound parameters

Abstract

The embodiments of the present specification provide a method and system for determining sound parameters, the method comprising: obtaining a video and an audio of a target space where a target sound is located through a camera and a microphone of a mobile device; obtaining a plurality of shooting positions based on the video; and determining the parameters of the target sound based on the plurality of shooting positions and the audio.

Description

Technical Field

[0001] The present invention relates to the field of audio, and in particular to a method and system for determining audio parameters. Background Art

[0002] With the rapid development of offline service industry, the application of audio is becoming more and more extensive. When using audio, users can adjust the audio parameters of the audio to get a better listening experience. The accuracy and intelligence of audio parameter adjustment will directly affect the user experience.

[0003] Therefore, it is necessary to provide a method and system for determining sound parameters to facilitate accurate determination of sound parameters. Summary of the invention

[0004] One of the embodiments of the present specification provides a method for determining sound parameters, the method comprising: obtaining a video and an audio of a target space where a target sound is located through a camera and a microphone of a mobile device; obtaining a plurality of shooting positions based on the video; and determining the parameters of the target sound based on the plurality of shooting positions and the audio.

[0005] One of the embodiments of the present specification provides a sound parameter determination system, the system comprising: a first acquisition module, used to acquire video and audio of a target space where a target sound is located through a camera and a microphone of a mobile device; a second acquisition module, used to acquire multiple shooting positions based on the video; and a determination module, used to determine the parameters of the target sound based on the multiple shooting positions and the audio.

[0006] One of the embodiments of the present specification provides a device for determining sound parameters, the device comprising: at least one storage device comprising a set of instructions; and at least one processor configured to communicate with the at least one storage device, wherein when the instructions are executed, the at least one processor is configured to instruct the system to perform operations, including: obtaining a video and audio of a target space where a target sound is located through a camera and a microphone of a mobile device; obtaining multiple shooting positions based on the video; and determining the parameters of the target sound based on the multiple shooting positions and the audio.

[0007] One of the embodiments of the present specification provides a computer-readable storage medium, including executable instructions. When executed by at least one processor, the executable instructions instruct the at least one processor to execute a method, the method comprising: obtaining a video and an audio of a target space where a target sound is located through a camera and a microphone of a mobile device; obtaining multiple shooting positions based on the video; and determining parameters of the target sound based on the multiple shooting positions and the audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0009] Figure 1 is a schematic diagram of an application scenario of a system for determining sound parameters according to some embodiments of this specification;

[0010] Figure 2 is an exemplary flow chart of a method for determining sound parameters according to some embodiments of this specification;

[0011] Figure 3 is an exemplary flow chart of determining multiple shooting positions according to some embodiments of this specification;

[0012] Figure 4 is an exemplary flow chart of a method for determining sound parameters according to other embodiments of this specification;

[0013] Figure 5 is an exemplary module diagram of a sound parameter determination system according to some embodiments of the present specification. DETAILED DESCRIPTION

[0014] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0015] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0016] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0017] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0018] Figure 1 It is a schematic diagram of an application scenario of the sound parameter determination system shown in some embodiments of this specification.

[0019] like Figure 1 As shown, the application scenario of the sound parameter determination system 100 may include a processor 110, a network 120, a storage device 130, a mobile device 140, and a sound device 150. In some embodiments, the application scenario also includes a scenario example 160.

[0020] The audio parameter determination system 100 can determine the audio parameters and adjust the audio parameters by implementing the methods and / or processes disclosed in this specification. In some embodiments, the audio parameter determination system 100 can be used in an audio-visual space where one or more audio devices are installed. For example, the audio parameter determination system 100 can be applied to KTV, cinemas, home theaters, etc. In some embodiments, the audio parameter determination system 100 can be used to determine the optimal parameters for the audio device to play according to the orientation of the audio device in the audio-visual space.

[0021] The processor 110 can be used to obtain data and / or transmit data. For example, the processor 110 can access information and / or data stored in the mobile device 140, the audio device 150 and / or the storage device 130 via the network 120. The processor 110 refers to a system with computing capabilities. In some embodiments, the processor 110 can be a single server or a server group. In some embodiments, the processor 110 can be local or remote. In some embodiments, the processor 110 can be implemented on a cloud platform. In some embodiments, the processor 110 can determine the audio parameters of the audio device 150 based on the audio obtained by the mobile device 140, the video obtained by the mobile device 140 and / or the displacement information of the mobile device 140. In some embodiments, the processor 110 can determine the audio parameters of the audio device 150 based on VR (Virtual Reality) technology.

[0022] The network 120 may include any suitable network that can be used for information and / or data exchange of the sound parameter determination system. In some embodiments, one or more components of the sound parameter determination system (e.g., the processor 110, the mobile device 140, the sound device 150, and the storage device 130, etc.) can exchange information and / or data with one or more components of the sound parameter determination system via the network 120.

[0023] The storage device 130 may store data and / or instructions. In some embodiments, the storage device 130 may store data obtained from the mobile device 140, the audio device 150, and / or the processor 110. For example, the storage device 130 may store a video obtained from the mobile device 140. For another example, the storage device 130 may store audio parameters in the audio device 150. In some embodiments, the storage device 130 may store data and / or instructions, and the processor 110 may execute or use the data and / or instructions to complete the exemplary methods described in this application.

[0024] In some embodiments, the storage device 130 may be connected to the network 120 to communicate with the processor 110, the mobile device 140, and / or the audio device 150. The processor 110, the mobile device 140, and / or the audio device 150 may access data or instructions stored in the storage device 130 via the network 120. In some embodiments, the storage device 130 may be part of the processor 110, the mobile device 140, and / or the audio device 150, or may be independent.

[0025] The mobile device 140 may refer to one or more terminal devices or software used by the user. In some embodiments, the mobile device 140 refers to a portable device with input and / or output functions. For example, the mobile device 140 may include a smart phone 141, a laptop computer 142, a tablet computer 143, a wearable device, and a smart mobile device, or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant (PDA), a gaming device, a navigation device, a handheld terminal (POS), or any combination thereof. In some embodiments, the mobile device 140 includes a camera and a microphone. The camera may be one of the components of a module that performs the video input function of the mobile device, and the microphone may be one of the components of a module that performs the audio input function in the mobile device. In some embodiments, the mobile device 140 has one of the devices with input and / or output functions, image acquisition functions, video acquisition functions, or audio acquisition functions, or any combination thereof. In some embodiments, the mobile device 140 may obtain a VR scene of the audio-visual space where the sound is located, and upload the VR scene to the processor 110 to determine multiple test positions and obtain multiple shooting positions. For more information about VR scene construction and obtaining shooting positions, please refer to Figure 4 Description of the part.

[0026] In some embodiments, the mobile device 140 can be used to obtain a video containing the audio device 150, and can also be used to obtain the audio played by the audio device 150. In some embodiments, the user can move in space while shooting a video to adjust the content of the video screen. In some embodiments, the mobile device 140 can be built with a motion sensor to obtain the user's motion information during the shooting process, such as the direction, speed, distance, etc. of movement.

[0027] The audio device 150 may be a system composed of devices that have the function of converting audio signals into sounds for playback. In some embodiments, the components of the audio device 150 may include power amplifiers, peripheral devices (including compressors, effectors, equalizers, VCDs, DVDs, etc.), speakers (speakers, speakers), mixers, microphones, display devices, etc. Figure 1 As shown, the audio device 150 may include other audio playback devices such as a speaker 151, an audio signal processing device 152, and a combined audio device 153. In some embodiments, the components of the audio device 150 may include a front speaker, a rear speaker, a center surround speaker, a subwoofer speaker, etc.

[0028] like Figure 1 In the illustrated scenario example 160, the home theater includes a mobile terminal 141, a rear speaker 151-1, a front speaker 151-2, and a subwoofer speaker 151-3. Exemplarily, the mobile terminal 1401 successively passes by the rear speaker 151-1 located next to the sofa, the front speaker 151-2 located next to the TV cabinet, and the subwoofer speaker 151-3 located above the TV, and determines the parameters of the speakers in the living room space based on the displacement change data of the mobile terminal 141 and the video and / or audio recorded by the mobile terminal 141 including the aforementioned speakers.

[0029] Figure 2 is an exemplary flow chart of a method for determining sound parameters according to some embodiments of the present specification. In some embodiments, process 200 may be implemented as a set of instructions (e.g., an application) stored in a storage device (e.g., storage device 130). A processing device (e.g., processor 110 and / or one or more modules) may execute the instruction set, and when executing the instructions, the processing device may be configured to perform process 200. The operations of the illustrated process presented below are intended to be illustrative. In some embodiments, process 200 may be accomplished using one or more additional operations not described and / or one or more operations not discussed. In addition, Figure 2 The order in which the operations of process 200 are shown and described below does not limit the scope of the present application.

[0030] In step 210, the processing device (eg, the first acquisition module 510) acquires the video and audio of the target space where the target sound is located through the camera and microphone of the mobile device.

[0031] The target sound refers to the sound equipment whose parameters are to be determined. In some embodiments, the target sound may include one or more speakers, or sound equipment arranged in various combinations. For example, the target sound may include one or more of a front speaker, a rear speaker, a center surround speaker, and a subwoofer speaker. For another example, the target sound may include a speaker and an audio signal processing device (such as an effector, a mixer, an equalizer, etc.). In some embodiments, the target sound can be used in various audio-visual spaces, such as a home theater, a professional audio-visual room, etc.

[0032] The target space refers to the audio-visual space where the target sound is placed, for example, a living room in a house, a waiting area in a restaurant, a screening room in a cinema, etc.

[0033] The video of the target space refers to the video shot in the target space, and the video can display information about the target space and / or objects therein. For example, the video can reflect information such as the position of the mobile device, the placement of the target audio in the target space, the position of other objects in the target space, the size of the target space, etc. In some embodiments, the video can have preset requirements, such as including audio, etc. In some embodiments, the video can be shot under the guidance of the mobile device, such as through voice or displaying text. In some embodiments, the video can be a continuously shot video, or it can be a splicing of multiple videos.

[0034] The audio of the target space refers to the audio recorded in the target space, and the audio can record the audio signal of the target speaker playing the sound. For example, the audio can reflect the relative position of the mobile device and the target speaker, the volume of the target speaker, the sound quality of the target speaker and other information. There is a corresponding relationship between the audio and the video. In some embodiments, the played sound can be a user input sound, such as music, broadcast, etc., or it can be a test sound dedicated to debugging the speaker. In some embodiments, the audio can be from one or more target speakers; if the audio comes from multiple target speakers, the audio can be played by multiple speakers simultaneously and / or each of the multiple speakers can be played separately.

[0035] Step 220: The processing device (eg, the second acquisition module 520) acquires a plurality of shooting positions according to the video.

[0036] The shooting position refers to the position information of the mobile device in the target space when the mobile device inputs video and / or audio, such as orientation, direction, etc. A shooting position corresponds to a frame of image in the video and / or a set of audio data in the audio. In some embodiments, the shooting position can be represented by the coordinate position of the spatial coordinate system of the target space, and the spatial coordinate system can be two-dimensional or three-dimensional. For example, in a spatial coordinate system based on a 10x 10x10 meter target space and with the southwest corner as the origin (0,0,0), if a mobile device is in the center of the target space and is held in front of the chest by the person being photographed, the shooting position of the mobile device can be (5,5,1.5). In some embodiments, the shooting position can be the relative position of the mobile device determined based on the position of the sound. For example, if the sound A is in the target space and its position is known, the shooting position can be "20 degrees south of the sound A, 3 meters away". The shooting position can also be the relative position of the mobile device determined by using other objects as reference objects. For example, the reference object can be an object whose position in the target space does not change, such as furniture, murals, etc.

[0037] In some embodiments, obtaining multiple shooting positions based on the video includes: extracting multiple frames of video images from the video according to a preset time interval, wherein two adjacent frames of video images in the multiple frames of video images have overlapping parts; and determining multiple shooting positions based on the multiple frames of video images.

[0038] The preset time interval refers to the time interval for extracting video images, such as 2 seconds, 1 second, etc. The preset time interval can be set according to actual needs. In some embodiments, the preset time interval can be changed according to the extracted video images. For example, when two adjacent frames of video images do not overlap, the preset time interval is shortened, and the images are continuously extracted based on the shortened preset time interval until the two frames of video images have overlapping parts.

[0039] The multi-frame video images refer to a part or all of the video images constituting the video. Each frame of the multi-frame video images corresponds to a shooting position.

[0040] The overlapping of two adjacent frames of video images in the multiple frames of video images means that the two frames of video images contain at least one identical, identifiable object. For example, audio, furniture, decorations, etc. The overlapping of two frames of video images can help obtain the relative change between the two shooting positions corresponding to the two frames of video images. For example, video image A corresponds to shooting position a, and its next adjacent frame of video image B corresponds to shooting position b. Both frames of images include audio C and the audio position c is known. Then, the position information ac of the shooting position a relative to the audio position c can be obtained through video image A, and the position information bc of the shooting position b relative to the audio position c can be obtained through video image B. Then, based on the two position information ac and bc, the position change of the shooting position b relative to the shooting position a can be obtained.

[0041] In some embodiments, the mobile device can guide the user's movement during shooting. In some embodiments, the guidance method can be a voice prompt, a text prompt, and / or a screen icon prompt, etc. In some embodiments, the guidance can be generated and / or changed according to the information received by the mobile device. For example, the guidance method and the content of the guidance can be generated and / or changed according to the location information, sound information, visual information, etc. received by the mobile device. Exemplarily, according to the location information received by the mobile device, if it is determined that the relative distance between the mobile device and the audio device increases, the content of the guidance corresponding to the mobile device (for example, the moving direction) can be adjusted accordingly to shorten the relative distance between the mobile device and the audio device to ensure the accuracy of the guidance. In some embodiments, the guidance can also be a simple and broad prompt, such as "move to the first audio", etc. In some embodiments, when the movement is too fast or other reasons cause two adjacent video images to have no overlapping parts, the mobile device can guide the user to return to the previous position and move again as required. In some embodiments, the mobile device can guide the user to move to one or more audio positions according to the sound volume emitted by the audio and the direction of the sound, or it can guide the user to move to the corresponding specified position according to the location information. The guidance can also be obtained by the shooting position or other methods based on the acquisition of multiple frames of video, or it can be realized by vibration or other methods. For more information about obtaining the shooting position based on multiple frames of video, please refer to Figure 3 .

[0042] In some embodiments, the obtaining of multiple shooting positions based on the video further includes: obtaining multiple shooting positions based on the video and the motion sensor of the mobile device. For example, during video shooting, the mobile device may first guide the user to move to a reference position (e.g., the sound position), and then obtain the displacement of the mobile device relative to the reference position in combination with the motion sensor to further obtain multiple shooting positions. In some embodiments, the mobile device may determine whether the user has reached the sound position and / or guide the user to approach the sound position based on the received sound information, for example, by judging and / or guiding based on the volume, sound direction, etc. For a more detailed description of the reference position, see Figure 3 and its related description.

[0043] In some embodiments, the processing device (e.g., the second acquisition module 520) can acquire multiple shooting positions based on the motion sensor. For example, the mobile device can first guide the user to move to a reference position (e.g., the position of the audio system), and then detect the displacement of the mobile device relative to the reference position through the motion sensor, thereby acquiring multiple shooting positions. In some embodiments, the mobile device can guide the user's movement by voice. In some embodiments, the mobile device can set the position where the motion sensor and / or microphone is activated as the reference position. For example, the reference position can be a position where the mobile device is placed within 0.1 meters in front of the audio system when the user activates the motion sensor and / or microphone.

[0044] A motion sensor refers to a module in a mobile device that can obtain displacement information. In some embodiments, the displacement information may include distance, direction, etc. In some embodiments, the displacement information may be calculated based on the detected acceleration and / or angular velocity.

[0045] In some embodiments, the processing device may combine the shooting position obtained based on the motion sensor with the shooting position obtained based on the video image to obtain a more accurate shooting position. In some embodiments, the combination method may be arithmetic average, weighted average, etc. For example, the shooting position obtained based on the motion sensor is 1 meter due north of the sound, and the shooting position obtained based on the video image is 1.1 meters due north of the sound. Then, the shooting position outputted according to the arithmetic average is 1.05 meters due north of the sound. For another example, if it is known from historical data, mobile device parameters and other information that the motion sensor of a mobile device is more accurate than its camera in terms of position determination, a higher weight, such as 0.8, may be given to the motion sensor, and then the weighted average is used to obtain the shooting position outputted as 1.02 meters due north of the sound. In some embodiments, the combination method may also use machine learning models and / or other algorithms.

[0046] Combining motion sensors to obtain the shooting position provides new considerations and parameters based on the above method of combining videos to obtain the shooting position, improves the accuracy of the results, and also provides additional solutions for obtaining the shooting position when the video is not ideal (for example, the environment is too dark, the camera is not clear, or the lens is damaged, etc.).

[0047] In step 230 , a processing device (eg, determination module 530 ) determines parameters of the target sound according to the multiple shooting positions and the audio.

[0048] In some embodiments, the processing device analyzes multiple shooting positions and their corresponding multiple frames of video images and multiple groups of audio data to identify the layout information of the target space and determine the parameters of the target sound according to the layout information.

[0049] Layout information refers to information related to objects arranged in the target space. In some embodiments, the layout information may include, but is not limited to, the position of the target sound, the distance between the shooting position and the target sound, the position of the objects placed in the target space, the material of the objects placed, the shape of the objects placed, the size of the objects placed, the size of the target space, the standing waves of the target space, the reverberation of the target space, the sensitivity of the target sound, the gain formula of the target sound, etc.

[0050] In some embodiments, the layout information of the target space can be identified in a variety of ways. In some embodiments, the layout information of the target space can be identified by a machine learning model and / or in combination with related algorithms. Among them, the machine learning model may include but is not limited to a convolutional neural network (CNN), a long short-term memory (LSTM) model, etc. Related algorithms may include but are not limited to LayoutNet, Flat2Layout, E2P, geometric algorithms, depth information algorithms, etc.

[0051] The position of the target sound refers to the placement position of the target sound in the target space. In some embodiments, the position of the target sound can be identified by image recognition technology. In some embodiments, the processing device can extract the feature of the target sound from the video image by a convolutional neural network, and the feature is used to determine the position of the target sound. For example, the processing device processes the feature by a neural network (NN) to determine the position of the target sound.

[0052] In some embodiments, the distance between the shooting position and the target sound can be identified in a variety of ways. In some embodiments, the distance between the shooting position and the target sound can be identified by image recognition technology. In some embodiments, the processing device can extract the features of the target sound and the shooting position from the video image through a convolutional neural network. The processing device determines the baseline between the shooting position and the target sound based on the above features. For example, the processing device processes the feature through a neural network to determine the baseline between the target sound and the shooting position. The processing device can determine the actual length corresponding to the baseline in the real scene in combination with the proportion of the video image, and determine the actual length as the distance between the shooting position and the target sound. In some embodiments, the distance between the shooting position and the target sound can also be obtained by calculating the coordinate difference between the two in the target space.

[0053] Placed objects refer to objects placed in the target space, such as furniture or decorations such as sofas, tables and chairs, curtains, and paintings. In some embodiments, the location of the placed objects can be identified by image recognition technology. In some embodiments, the processing device can extract the features of the placed objects from the video image through a convolutional neural network, and the features are used to determine the location of the placed objects. For example, the processing device processes the features through a neural network to determine the location of the placed objects. In some embodiments, the convolutional neural network can extract the features of the placed objects through a preset algorithm. For example, the SSD algorithm (Single Shot MultiBox Detector).

[0054] The material, shape and size of the placed objects can affect the parameters of the target sound. For example, the placed objects made of sound-absorbing materials can absorb the sound waves of the target sound, thereby reducing the sound quality and volume of the target sound. For another example, the placed objects with irregular shapes or large sizes will hinder the transmission of the sound waves of the target sound, thereby reducing the surround sound effect and stereo effect of the target sound.

[0055] Standing waves refer to the composite waves formed by the interference of two sinusoidal waves with the same wavelength, period, frequency and wave speed moving in opposite directions. The standing waves in the target space will weaken some of the sound waves emitted by the target sound, thereby reducing the user's listening experience. The standing waves in the target space are related to the size of the target space. For example, the larger the size of the target space, the smaller the critical frequency of the standing waves in the target space, and the smaller the impact of the standing waves on the target sound. In some embodiments, the standing wave is represented by the standing wave ratio obtained by devices such as couplers, detectors, single-chip microcomputers and / or operational amplifier circuits. For example, a wide-band bidirectional coupler is connected in series at the multi-frequency signal combining port, and the input power and reflected power are extracted respectively and sent to the detector, and the RF signal power is converted into a DC signal, and then the single-chip microcomputer or a simple operational amplifier circuit is used for calculation to obtain the standing wave ratio of the port, which represents the standing wave of the port.

[0056] Reverberation refers to the acoustic phenomenon that sound continues to exist after the sound source stops producing sound. The reverberation of the target space will affect the sound quality of the target sound. The reverberation of the target space is related to the size of the target space and the material of the target space. In some embodiments, the reverberation can be obtained by a reverberation test device. The reverberation test device may include a microphone, a sound calibrator and / or a vibration noise signal acquisition system, etc.

[0057] The sensitivity of the target speaker refers to the signal voltage at the input end when the speaker's power amplifier reaches full power output. The larger the signal voltage, the lower the sensitivity. The sensitivity of the speaker is usually used to reflect the sound volume perceived by the human ear. The higher the sensitivity of the speaker, the louder the user perceives the speaker to be. However, too high a sensitivity will damage the sound quality of the speaker. Therefore, the sensitivity of the speaker should be controlled within a reasonable range to provide users with a high-quality listening experience.

[0058] The gain formula of the target sound refers to the formula used in the process of determining the gain of the target sound. In some embodiments, the gain formula of the target sound may be a formula for determining the maximum voltage capacity of the target sound. The target sound may be protected by determining the maximum voltage capacity of the target sound.

[0059] In some embodiments, the parameters of the target sound may include but are not limited to the gain of the target sound, the output power of the target sound, the delay of the target sound, etc.

[0060] The gain of the target sound refers to the signal amplification rate of the target sound. For example, the amplification rate of a smaller output voltage after being amplified by an amplifier to become a larger output voltage. In some embodiments, the processing device determines multiple candidate gains of the target sound, and determines the gain of the target sound from the multiple candidate gains. The gain of the target sound can determine whether the target sound has a better sound quality improvement at the same volume. In some embodiments, the gain of the target sound can be determined and adjusted by a corresponding equalizer (EQ).

[0061] The output power of the target sound refers to the rated power of the target sound when it is used. The output power of the target sound can determine the maximum sound intensity of the target sound. In some embodiments, the output power of the target sound can be determined based on the analysis results of the video. In some embodiments, the analysis results of the video can include the volume of the target space. In some embodiments, the optimal output power of the target sound can be determined based on the optimal correspondence between the volume of the target space and the output power of the target sound. For example, when the volume of the target space is 20m 3 When , the optimal output power of the target sound can be determined as 60W according to the optimal corresponding relationship.

[0062] The delay of the target sound refers to the delay for the user to receive the sound emitted by each sound. Appropriate delay can improve the stereo sound quality of the target sound. For example, when the delay between two sound sources is 5ms to 35ms, the human ear can only feel the existence of one sound source ahead; when the delay of the sound source is 30ms to 50ms, the human ear can roughly distinguish the existence of the two sound sources; when the delay of the sound source is greater than 50ms, the human ear can feel the existence of two sound sources at the same time. The smaller the delay of the target sound, the softer the sound quality of the target sound; the larger the delay of the target sound, the stronger the stereo surround sense of the sound quality of the target sound. The delay data of the target sound can be determined by an audio device (for example, an audio delay device) that performs delay processing on the sound signal.

[0063] In some embodiments, the parameters of the target sound can be determined in a variety of ways. In some embodiments, the parameters of the target sound can be determined based on a machine learning model. In some embodiments, the machine learning model may include but is not limited to a convolutional neural network, a long short-term memory model, etc. In some embodiments, the parameters of the target sound can be determined by the initial parameters of the target sound. In some embodiments, the initial parameters of the target sound may include but are not limited to one or more of the gain of the target sound, the output power of the target sound, and the delay of the target sound. In some embodiments, the parameters of the target sound can be determined based on the initial parameters of the target sound and / or the analysis results of the video. In some embodiments, multiple images or panoramic images of the target space can be input into the machine learning model, and the parameters of the target sound with better performance can be output. In some embodiments, the processing device can obtain the original parameters of the target sound stored in the storage device, and evaluate the sound quality of the target sound under the original parameters through an auxiliary device or program (e.g., a sound quality evaluation application), and determine the parameters of the optimized target sound.

[0064] By using videos and audio captured by mobile devices, users can more easily determine the parameters of the target audio, making professional parameter adjustments smarter and simpler, thereby providing users with better listening effects.

[0065] Figure 3 is an exemplary flow chart of determining multiple shooting positions according to some embodiments of the present specification. The process 300 includes the following steps.

[0066] Step 301: A processing device (eg, a first acquisition module 510) determines a first image in the plurality of video frames by using an image recognition model, wherein the first image includes a target sound.

[0067] An image recognition model refers to any model that has the function of recognizing an object in an image. In some embodiments, the image recognition model may be a machine learning model. In some embodiments, the image recognition model may include but is not limited to a convolutional neural network, a long short-term memory model, etc. In some embodiments, the input of the image recognition model may be the multiple frames of video images, and the output may be one or more frames of the multiple frames of video images that include the target sound, i.e., the first image.

[0068] Step 302: A processing device (eg, a second acquisition module 520) determines a reference position according to the first image.

[0069] The reference position refers to a reference position that guides the shooting position. For example, the shooting position can be described by the relative position of the reference position. In some embodiments, the reference position can be the position of the target sound in the target space. The position of the target sound in the target space can be determined based on the first image containing the target sound. For example, the position of the target sound in the target space is determined based on the distance and angle of the sound relative to the camera in the image. In some embodiments, the reference position can also be any object placed in the target space, such as furniture and decorations. In some embodiments, the reference position can also be a preset known position.

[0070] Step 303: The processing device (eg, the second acquisition module 520) determines a plurality of shooting positions according to the reference position and a plurality of frames of video images.

[0071] In some embodiments, the processing device determines the shooting position of the video image containing the reference object according to the reference position, and determines the shooting position of the adjacent video image according to the overlapping part in the adjacent video images. In some embodiments, the overlapping part of two adjacent video images can be used to determine the change in angle and distance between the corresponding two shooting positions. For example, there are three video images A, B, and C, where image A contains a speaker and a sofa, which is the first image and corresponds to the shooting position a; image B contains a sofa and a mural, which corresponds to the shooting position b; image C contains only the mural, which corresponds to the shooting position c. If the speaker is set as a reference object, the speaker position, that is, the reference position, and the shooting position a relative to the speaker position can be determined according to the distance and angle of the speaker from the camera in image A, for example, the shooting position a can be 1 meter due north of the speaker position. Image B and image A contain the overlapping part of the sofa, and the shooting position b relative to the shooting position a and / or relative to the speaker position can be determined according to the change in the angle and distance of the sofa in the two frames of images. For example, according to the change of the sofa, it is determined that the shooting position b is 2 meters due south of the shooting position a, and the shooting position b can be 1 meter due south of the speaker position. Similarly, image C and image B include an overlapping portion of the mural. Based on the changes in the angle and distance of the mural in the two frames of images, shooting position c relative to shooting position b and / or relative to the audio position can be determined. For example, based on the changes in the mural, if shooting position c is 1 meter due west of shooting position b, then shooting position c can be 45 degrees and 1.41 meters south-west of the audio position.

[0072] By determining multiple shooting positions based on the image recognition model, reference positions and overlapping parts, the relative position relationship between every two adjacent shooting positions can be established quickly and conveniently without consuming additional computing power to obtain the absolute position of each shooting position, thereby improving efficiency.

[0073] Figure 4 is an exemplary flow chart of a method for determining sound parameters according to other embodiments of this specification.

[0074] like Figure 4 As shown, the process 400 includes the following steps. In some embodiments, the process 400 may be performed by the determination module 530 .

[0075] Step 410: construct a VR scene of the target space according to the video.

[0076] A VR scene refers to a scene that is generated by real-life shooting and image processing to reconstruct a real object in three dimensions. For example, a VR scene of a target space may be a scene constructed by image processing and splicing based on a video of the target space. In some embodiments, VR scene production may include real-life shooting, image processing, and splicing. Equipment used for real-life shooting includes a SLR camera, a panoramic gimbal, a fisheye lens, a tripod, and / or an aircraft, etc. The splicing and processing of images may be performed in various feasible ways.

[0077] Step 420: Determine multiple test locations based on the VR scene.

[0078] The test position refers to the position for listening to the target sound obtained by calculation or other methods. In some embodiments, the test position may be the best position for listening to the target sound in theory. In some embodiments, the test position may also be the sound sample collection position that can optimize the sound parameter adjustment capability. For example, in a target space with very complex shapes and materials, calculation alone may not be able to achieve sufficient confidence, so sound samples can be collected at a specific test position to supplement and verify the calculation results, thereby optimizing the sound parameter adjustment capability.

[0079] In some embodiments, multiple test positions can be calculated based on the VR scene, the sound position and / or the layout information. In some embodiments, the sound position and / or the layout information can be calculated by Figure 2 The methods shown and their related descriptions are available and can also be obtained through other channels.

[0080] In some embodiments, the user can input a candidate test location in the VR scene. For example, the user can input the location of a commonly used sofa as a candidate test location. In some embodiments, the candidate test location can be used as a reference for calculating the test location. For example, when multiple similar test locations are calculated, a test location that is closer to the candidate test location can be given priority. In some embodiments, the candidate test location can be directly used as a test location.

[0081] Step 430 , obtaining multiple shooting positions by guiding the user to move the mobile device to the multiple test positions.

[0082] The process of the user shooting videos and / or recording audios at the multiple test locations with a mobile device is the process of converting the multiple test locations into the multiple shooting locations. In some embodiments, the calculated multiple test locations can be displayed through a VR scene, and / or the user can be guided to move the mobile device to the test location through the VR scene.

[0083] Figure 5 is an exemplary module diagram of a sound parameter determination system according to some embodiments of the present specification.

[0084] In some embodiments, the module 500 of the acoustic parameter determination system may include a first acquisition module 510 , a second acquisition module 520 , and a determination module 530 .

[0085] In some embodiments, the first acquisition module 510 may acquire the video and audio of the target space where the target sound is located through the camera and microphone of the mobile device.

[0086] In some embodiments, the second acquisition module 520 can acquire multiple shooting positions according to the video. For instructions on acquiring multiple shooting positions, see Figure 2 Description of the corresponding content.

[0087] In some embodiments, the second acquisition module 520 can extract multiple frames of video images from the video according to a preset time interval, and determine multiple shooting positions according to the multiple frames of video images, wherein two adjacent frames of video images in the multiple frames of video images have overlapping parts.

[0088] In some embodiments, the second acquisition module 520 can determine the first image in the multiple frames of video images through an image recognition model; determine a reference position based on the first image; and determine the multiple shooting positions based on the reference position and the multiple frames of video images. The first image includes the target sound. The image recognition model and its related description can be found in Figure 3 Description of the corresponding content.

[0089] In some embodiments, the second acquisition module 520 may acquire the multiple shooting positions according to the video and a motion sensor of the mobile device.

[0090] In some embodiments, the second acquisition module 520 may also construct a VR scene of the target space according to the video; determine multiple test positions according to the VR scene; and obtain the multiple shooting positions by guiding the user to move the mobile device to the multiple test positions. Figure 4 Description of the corresponding content.

[0091] In some embodiments, the determination module 530 is configured to determine the parameters of the target sound according to the multiple shooting positions and the audio.

[0092] It should be understood that Figure 5The system and its modules shown can be implemented in various ways. For example, in some embodiments, the first acquisition module 510 can be used to acquire the video and audio of the target space where the target sound is located through the camera and microphone of the mobile device. The second acquisition module 520 can be used to acquire multiple shooting positions based on the video. The determination module can be used to determine the parameters of the target sound based on the multiple shooting positions and the audio.

[0093] It should be noted that the above description of the above system and its modules is only for the convenience of description and does not limit the present specification to the scope of the embodiments. It is understandable that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the modules or form a subsystem to connect with other modules without deviating from this principle. In some embodiments, Figure 5 The first acquisition module 510, the second acquisition module 520 and the determination module 530 disclosed in the specification can be different modules in a system, or one module can realize the functions of two or more modules. For example, each module can share a storage module, or each module can have its own storage module. Such variations are all within the protection scope of this specification.

[0094] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0095] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0096] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0097] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0098] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0099] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0100] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A method for determining an acoustic parameter, comprising: The video and audio of the target space where the target sound is located are obtained through the camera and microphone of the mobile device; Extracting multiple frames of video images from the video according to a preset time interval, wherein two adjacent frames of video images in the multiple frames of video images have overlapping parts; Determining a plurality of first initial shooting positions according to the plurality of frames of video images; Acquire a plurality of second initial shooting positions according to the video and a motion sensor of the mobile device; Determine a plurality of shooting positions based on a combination of the plurality of first initial shooting positions and the plurality of second initial shooting positions, wherein the combination method includes at least an arithmetic average and a weighted average; Analyze the multiple shooting positions and the multiple frames of video images and the audio corresponding thereto to determine layout information of the target space; Determining parameters of the target sound according to the layout information, wherein the multiple frames of video images include a first image, and determining multiple first initial shooting positions according to the multiple frames of video images includes: Determine the first image in the multiple frames of video images by using an image recognition model, wherein the first image includes the target sound; determining a reference position according to the first image; Determining the plurality of first initial shooting positions according to the reference position and the plurality of frames of video images, wherein acquiring the plurality of second initial shooting positions according to the video and a motion sensor of the mobile device comprises: During the video shooting process, the mobile device guides the user to move to the reference position; The plurality of second initial shooting positions are acquired according to the displacement of the mobile device relative to the reference position detected by the motion sensor.

2. The method according to claim 1, wherein the layout information includes a distance between a shooting position and the target sound; The determining the layout information of the target space includes: Extracting features of the target sound and the shooting location from the multiple frames of video images by using a convolutional neural network; determining a baseline between the shooting position and the target sound according to the feature; Determining an actual length of the baseline in the real scene based on the ratio of the multiple frames of video images; The actual length is determined as the distance between the shooting position and the target sound.

3. The method according to claim 1, further comprising: According to the video, construct a VR scene of the target space; According to the VR scene, determining a plurality of test locations; The multiple shooting positions are obtained by guiding the user to move the mobile device to the multiple test positions.

4. A system for determining acoustic parameters, the system comprising: The first acquisition module is used to acquire the video and audio of the target space where the target sound is located through the camera and microphone of the mobile device; A second acquisition module is used to extract multiple frames of video images from the video according to a preset time interval, wherein two adjacent frames of video images in the multiple frames of video images have overlapping parts; Determine a plurality of first initial shooting positions according to the plurality of frames of video images; obtain a plurality of second initial shooting positions according to the video and a motion sensor of the mobile device; Determine a plurality of shooting positions based on a combination of the plurality of first initial shooting positions and the plurality of second initial shooting positions, wherein the combination method includes at least an arithmetic average and a weighted average; a determination module, configured to analyze the plurality of shooting positions and the plurality of frames of video images and the audio corresponding thereto, determine layout information of the target space; and determine parameters of the target audio according to the layout information, wherein the plurality of frames of video images include a first image, and determining the plurality of first initial shooting positions according to the plurality of frames of video images includes: Determine the first image in the multiple frames of video images by using an image recognition model, wherein the first image includes the target sound; determining a reference position according to the first image; Determining the plurality of first initial shooting positions according to the reference position and the plurality of frames of video images, wherein acquiring the plurality of second initial shooting positions according to the video and a motion sensor of the mobile device comprises: During the video shooting process, the mobile device guides the user to move to the reference position; The plurality of second initial shooting positions are acquired according to the displacement of the mobile device relative to the reference position detected by the motion sensor.

5. A device for determining an acoustic parameter, the device comprising: at least one storage device including a set of instructions; as well as at least one processor configured to communicate with the at least one storage device, wherein when executing the instructions, the at least one processor is configured to instruct the apparatus to perform operations comprising: The video and audio of the target space where the target sound is located are obtained through the camera and microphone of the mobile device; Extracting multiple frames of video images from the video according to a preset time interval, wherein two adjacent frames of video images in the multiple frames of video images have overlapping parts; Determining a plurality of first initial shooting positions according to the plurality of frames of video images; Acquire a plurality of second initial shooting positions according to the video and a motion sensor of the mobile device; Determine a plurality of shooting positions based on a combination of the plurality of first initial shooting positions and the plurality of second initial shooting positions, wherein the combination method includes at least an arithmetic average and a weighted average; Analyze the multiple shooting positions and the multiple frames of video images and the audio corresponding thereto to determine layout information of the target space; Determining parameters of the target sound according to the layout information, wherein the multiple frames of video images include a first image, and determining multiple first initial shooting positions according to the multiple frames of video images includes: Determine the first image in the multiple frames of video images by using an image recognition model, wherein the first image includes the target sound; determining a reference position according to the first image; Determining the plurality of first initial shooting positions according to the reference position and the plurality of frames of video images, wherein acquiring the plurality of second initial shooting positions according to the video and a motion sensor of the mobile device comprises: During the video shooting process, the mobile device guides the user to move to the reference position; The plurality of second initial shooting positions are acquired according to the displacement of the mobile device relative to the reference position detected by the motion sensor.

6. A non-transitory computer-readable medium comprising executable instructions, which, when executed by at least one processor, instruct the at least one processor to perform a method comprising: The video and audio of the target space where the target sound is located are obtained through the camera and microphone of the mobile device; Extracting multiple frames of video images from the video according to a preset time interval, wherein two adjacent frames of video images in the multiple frames of video images have overlapping parts; Determining a plurality of first initial shooting positions according to the plurality of frames of video images; Acquire a plurality of second initial shooting positions according to the video and a motion sensor of the mobile device; Determine a plurality of shooting positions based on a combination of the plurality of first initial shooting positions and the plurality of second initial shooting positions, wherein the combination method includes at least an arithmetic average and a weighted average; Analyze the multiple shooting positions and the multiple frames of video images and the audio corresponding thereto to determine layout information of the target space; Determining parameters of the target sound according to the layout information, wherein the multiple frames of video images include a first image, and determining multiple first initial shooting positions according to the multiple frames of video images includes: Determine the first image in the multiple frames of video images by using an image recognition model, wherein the first image includes the target sound; determining a reference position according to the first image; Determining the plurality of first initial shooting positions according to the reference position and the plurality of frames of video images, wherein acquiring the plurality of second initial shooting positions according to the video and a motion sensor of the mobile device comprises: During the video shooting process, the mobile device guides the user to move to the reference position; The plurality of second initial shooting positions are acquired according to the displacement of the mobile device relative to the reference position detected by the motion sensor.

Images