Method and device for realizing visual and auditory synchronous perception

Abstract

The invention provides a method and device for realizing visual and auditory synchronous perception, and the method comprises the steps: obtaining the relative position information between a played video image and an audience and the type of a room where the audience is located under the condition that a video application is started; determining a volume adjustment gain based on the acquired relative position information and the type of the room where the audience is located; adjusting the sound to be played of the video application according to the volume adjustment gain; when the video is played, the relative orientation and distance between the image and the audience can be obtained, the type of the room can be judged, the audio signal to be played is dynamically adjusted according to the orientation and distance between the audience and the image and the type of the room, and the virtual sound image synchronously changing with the image is generated. The audience can obtain the same spatial perception in auditory sense and visual sense, and the immersion of video application is increased.

Description

Technical Field

[0001] The present application relates to the technical field of audio and video signal processing, and in particular to a method and device for realizing synchronous visual and auditory perception. Background Art

[0002] In video experience scenarios such as watching movies, the audience can get a better audio and video experience. However, in scenarios such as video conferencing, video lectures, and remote teaching, the direction and size of the sound perceived by the audience through the audio and video devices are often fixed and do not change with the image on the screen. The former produces an auditory perception of fixed position and distance, while the latter produces visual dynamic changes. The lack of synchronization between the two will cause auditory fatigue and to some extent cause a sense of separation between sound and picture in space. Therefore, a method and device for achieving synchronous visual and auditory perception is needed, which can dynamically adjust the sound following the changes in the visual sound source image, so that the audience can obtain auditory synchronized distance and direction perception and increase the sense of immersion. Summary of the invention

[0003] The present application shows a method and device for achieving synchronous visual and auditory perception.

[0004] In a first aspect, the present application provides a method for achieving synchronous visual and auditory perception, the method comprising:

[0005] When a video application is turned on, the relative position information between the played video image and the viewer and the type of room the viewer is in are obtained;

[0006] Determining a volume adjustment gain based on the acquired relative position information and the type of room where the audience is located;

[0007] The sound to be played of the video application is adjusted according to the volume adjustment gain.

[0008] According to the method of the first aspect of the present application, based on the optical reversibility principle, the video camera is compared to the audience, and the relative position information between the played video image and the audience is obtained through the video acquisition end.

[0009] According to the method of the first aspect of the present application, based on the parallax principle, multiple cameras arranged at the audience's location are used to obtain the relative position information between the played video image and the audience.

[0010] According to the method of the first aspect of the present application, the relative position information between the played video image and the viewer is determined by combining the viewer's orientation and angle in front of the display screen with the viewer's imaging position and screen depth on the display screen.

[0011] According to the method of the first aspect of the present application, determining the volume adjustment gain based on the acquired relative position information and the type of room where the audience is located specifically includes:

[0012] Determine a corresponding sound pressure attenuation function based on the type of room where the audience is located;

[0013] Substituting the distance value of the current video image frame and the distance value of the previous video image frame contained in the relative position information into the sound pressure attenuation function to calculate the sound pressure difference value;

[0014] When the sound pressure difference is greater than a preset threshold, the sound pressure difference is converted to obtain the volume adjustment gain.

[0015] According to the method of the first aspect of the present application, the corresponding sound pressure attenuation function is determined based on the type of room where the audience is located, specifically:

[0016] When the room type where the audience is located is a large space or soft decoration, the sound pressure attenuation function is a linear function;

[0017] When the room type where the audience is located is a long and narrow room, the sound pressure attenuation function is a logarithmic function;

[0018] When the room type where the audience is located is a small room, the sound pressure attenuation function is an inverse logarithmic function.

[0019] According to the method of the first aspect of the present application, the sound to be played of the video application is adjusted according to the volume adjustment gain, specifically: the volume adjustment gain is applied to the sound to be played, and a virtual sound and image generation algorithm is used to drive multiple speakers to sound, so as to realize the virtual sound and image generation of the azimuth angle of the corresponding video image.

[0020] In a second aspect, the present application shows a device for realizing synchronous perception of vision and hearing, the device comprising:

[0021] The first processing module is configured to, when the video application is turned on, obtain the relative position information between the played video image and the viewer and the type of room the viewer is in;

[0022] A second processing module is configured to determine a volume adjustment gain based on the acquired relative position information and the type of room where the audience is located;

[0023] The third processing module is configured to adjust the sound to be played of the video application according to the volume adjustment gain.

[0024] According to the device of the second aspect of the present application, based on the optical reversibility principle, the video camera is compared to the audience, and the relative position information between the played video image and the audience is obtained through the video acquisition end.

[0025] According to the device of the second aspect of the present application, based on the parallax principle, multiple cameras arranged at the position of the audience are used to obtain the relative position information between the played video image and the audience.

[0026] According to the device of the second aspect of the present application, the relative position information between the played video image and the viewer is determined by combining the viewer's orientation and angle in front of the display screen with the viewer's imaging position and screen depth on the display screen.

[0027] According to the device of the second aspect of the present application, the second processing module is specifically configured as follows:

[0028] Determine a corresponding sound pressure attenuation function based on the type of room where the audience is located;

[0029] Substituting the distance value of the current video image frame and the distance value of the previous video image frame contained in the relative position information into the sound pressure attenuation function to calculate the sound pressure difference value;

[0030] When the sound pressure difference is greater than a preset threshold, the sound pressure difference is converted to obtain the volume adjustment gain.

[0031] According to the device of the second aspect of the present application, when the type of the room where the audience is located is a large space or soft decoration, the sound pressure attenuation function is a linear function;

[0032] When the room type where the audience is located is a long and narrow room, the sound pressure attenuation function is a logarithmic function;

[0033] When the room type where the audience is located is a small room, the sound pressure attenuation function is an inverse logarithmic function.

[0034] According to the device of the second aspect of the present application, the third processing module is specifically configured to: apply the volume adjustment gain to the sound to be played, and use a virtual sound and image generation algorithm to drive multiple speakers to sound, so as to achieve virtual sound and image generation at the azimuth angle of the corresponding video image.

[0035] In a third aspect, the present application shows an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method for achieving synchronous visual and auditory perception as described in any of the above aspects.

[0036] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method for achieving synchronous visual and auditory perception as described in any of the above aspects.

[0037] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method for achieving synchronous visual and auditory perception as described in any of the above aspects.

[0038] The technical solution provided by this application may have the following beneficial effects:

[0039] The method of realizing synchronous visual and auditory perception of the present application obtains the relative position and distance between the image and the audience and determines the type of room when the video is played, and dynamically adjusts the audio signal to be played according to the position and distance between the audience and the image and the type of room, and generates a virtual sound and image that changes synchronously with the image, so that the audience can obtain the same spatial perception in hearing and vision, and increase the sense of immersion in video applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a step flow of a method for realizing synchronous visual and auditory perception in this application Figure 1 .

[0041] Figure 2 It is a schematic diagram of the structure of the solution combining measurement and 3D display in this application.

[0042] Figure 3 The following is a step flow of a method for realizing synchronous visual and auditory perception in this application: Figure 2 .

[0043] Figure 4 This is a structural block diagram of a device for realizing synchronous visual and auditory perception in the present application.

[0044] Figure 5 It is a block diagram of an electronic device of the present application.

[0045] Figure 6 It is a block diagram of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] Reference Figure 1 , shows a flowchart of the steps of a method for realizing synchronous visual and auditory perception of the present application, which can be applied to electronic devices, wherein the method specifically may include the following steps:

[0048] Step 101: When a video application is turned on, obtain relative position information between the played video image and the viewer and the type of room the viewer is in;

[0049] Step 102: determining a volume adjustment gain based on the acquired relative position information and the type of room where the audience is located;

[0050] Step 103: adjusting the sound to be played of the video application according to the volume adjustment gain.

[0051] In step 101, when a video application is turned on, the relative position information between the played video image and the viewer and the type of room where the viewer is located are obtained.

[0052] It should be noted that the video application in this embodiment refers to a video conferencing application, a video presentation application, a remote teaching application, etc., and this application does not make any specific limitation on this.

[0053] In this embodiment, when the video application is turned on, the audience will receive real-time video images and audio signals. When there are more than one participating audience and each audience is in a different position, the direction and size of the sound do not change with the image on the screen, that is, the vision and hearing are not synchronized, which will cause auditory fatigue and to some extent cause a sense of separation between the sound and the picture in space. Therefore, in this application, the relative position information between the played video image and the audience and the type of room the audience is in are first obtained, so as to adjust the playback of the audio signal according to the relative position information and the type of room the audience is in.

[0054] It should be noted that, in this embodiment, the relative position information includes the relative position and distance between the viewer and the video image.

[0055] In some embodiments, the method of realizing synchronous visual and auditory perception of the present application is based on the principle of optical reversibility, and the video camera is compared to the audience, and the relative position information between the played video image and the audience is obtained through the video acquisition end.

[0056] In some embodiments, the method of realizing synchronous visual and auditory perception of the present application is based on the principle of parallax, and uses multiple cameras arranged at the position of the audience to obtain the relative position information between the played video image and the audience.

[0057] In some embodiments, the method of achieving synchronous visual and auditory perception of the present application combines the orientation and angle of the viewer in front of the display screen with the imaging position and screen depth of the viewer on the display screen to determine the relative position information between the played video image and the viewer.

[0058] Specifically, this application proposes three solutions for obtaining the relative position and distance between the audience and the video image:

[0059] Solution 1: Based on the optical reversibility principle, the video camera is compared to the audience, and the video acquisition end measures the relative position and distance between the audience and the video image:

[0060] The video acquisition end uses optical, ultrasonic positioning or other methods to realize the speaker's position perception and calculate the position and distance between the speaker and the video acquisition camera. The calculated position and distance are synchronously encoded with the video data collected by the video camera and then transmitted or stored over the network.

[0061] It should be noted that the sampling frequency of the direction and distance is the same as the number of video frames.

[0062] Solution 2: Multiple cameras are arranged around the audience to obtain multiple frames of images, and the relative position and distance between the audience and the video image are measured based on the multi-frame images using the parallax principle:

[0063] For example, multiple cameras can be placed behind the audience to measure the image position and distance on the display screen through the principle of parallax.

[0064] Of course, multiple cameras can also be placed behind and on the sides of the audience to measure the image orientation and distance on the display screen using the parallax principle.

[0065] It should be noted that the measurement frequency is ≥ 20 Hz.

[0066] Solution three, a solution combining measurement and 3D display: The relative position information between the played video image and the audience is determined by combining the audience's position and angle in front of the display screen with the audience's imaging position and screen depth on the display screen.

[0067] The audience's position and distance in front of the display screen are perceived through optical, ultrasonic positioning or other means, and the display position information of the image is provided by the 3D display module. The two are combined to determine the relative distance and direction between the audience and the image at the image frame rate.

[0068] by Figure 2 To elaborate on the example:

[0069] Using ultrasonic positioning, measure the distance L of the audience in front of the screen and the coordinates of the intersection point on the screen corresponding to L;

[0070] The coordinates of the image on the screen are obtained by the 3D display module, and the horizontal distance W and the height distance H between the viewer and the image are calculated in combination with the coordinates of the intersection point corresponding to L;

[0071] The 3D display module obtains the screen depth of the image, i.e., the distance l in front of the screen;

[0072] The distance from the viewer to the 3D image and the horizontal azimuth and altitude azimuth are calculated based on trigonometric functions.

[0073] In some embodiments, the room type in the method for realizing synchronous visual and auditory perception of the present application is a large space, soft decoration, narrow and long or a small room. Of course, it can also be other types, which are not listed one by one here.

[0074] In this embodiment, after the user starts a video conference, video lecture, remote teaching or other application, the room type is determined by optical, ultrasonic or other methods. For example, the size information of the room is obtained by ultrasonic method, and then the room type (large space, small room or narrow and long room, etc.) is determined according to the size information; or the three-dimensional information of the room is obtained by multiple cameras, and then the room type (large space, small room or narrow and long room, etc.) is determined according to the three-dimensional information, and the decoration (soft decoration) of the room is identified by image information.

[0075] In step 102, a volume adjustment gain is determined based on the acquired relative position information and the type of room where the audience is located.

[0076] In some embodiments, see Figure 3 As shown, step S102 in the method for realizing synchronous visual and auditory perception of the present application specifically includes:

[0077] Determine a corresponding sound pressure attenuation function based on the type of room where the audience is located;

[0078] Substituting the distance value of the current video image frame and the distance value of the previous video image frame contained in the relative position information into the sound pressure attenuation function to calculate the sound pressure difference value;

[0079] When the sound pressure difference is greater than a preset threshold, the sound pressure difference is converted to obtain the volume adjustment gain.

[0080] It should be noted that the sound pressure attenuation function in this embodiment is a function of the relationship between the attenuation of sound and distance, and the sound pressure attenuation function may be a linear function, a logarithmic function, an inverse logarithmic function, or the like.

[0081] In some embodiments, in the method for realizing synchronous visual and auditory perception of the present application, when the type of the room where the audience is located is a large space or soft decoration, the sound pressure attenuation function is a linear function;

[0082] When the room type where the audience is located is a long and narrow room, the sound pressure attenuation function is a logarithmic function;

[0083] When the room type where the audience is located is a small room, the sound pressure attenuation function is an inverse logarithmic function.

[0084] It should be noted that the above correspondence between the room type and the sound pressure attenuation function is only an example, and there may be other correspondences, which are not listed here one by one.

[0085] In some embodiments, the method of realizing synchronous visual and auditory perception of the present application defines the distance value corresponding to the current video image frame as d1, and the distance value corresponding to the previous video image frame as d2. The two distance values ​​are substituted as parameters into the above-mentioned sound pressure attenuation function and the difference is calculated as the sound pressure difference △SPL.

[0086] In some embodiments, if the volume is adjusted for each sound pressure difference △SPL in the method of realizing synchronous visual and auditory perception of the present application, on the one hand, the consumption of hardware resources such as computing power is increased, and on the other hand, too small sound pressure difference will not be perceived by the human ear. Therefore, a preset threshold value A is set according to the hearing characteristics of the human ear, and the volume adjustment program will only be triggered when △SPL is greater than the preset threshold value.

[0087] For example, the preset threshold value A in the method for realizing synchronous visual and auditory perception of the present application can be set to 1 dB. Of course, it can also be set to other values, which are not listed here one by one.

[0088] It should be noted that the above is to calculate the difference and set the threshold from the perspective of sound pressure level. Optionally, the difference and threshold can also be calculated from the perspective of loudness. For example, the sound pressure difference △SPL obtained by substituting different frequency sounds into the above sound pressure attenuation function can be substituted into the A-weighted network to calculate the loudness level difference to simplify the calculation and save hardware resources. Exemplarily, the weighted frequency point is the 1 / 3 octave frequency point in the 20-20kHz frequency band, the weighting network is A-weighted, and the threshold of the loudness level difference is 1 square. Generally, the loudness level difference is smaller than the sound pressure level difference, so setting the threshold from the perspective of loudness can further reduce computing power consumption.

[0089] In step 103, the sound to be played of the video application is adjusted according to the volume adjustment gain.

[0090] In some embodiments, step S103 in the method of realizing synchronous visual and auditory perception of the present application is specifically: applying the volume adjustment gain to the sound to be played, and using a virtual sound and image generation algorithm to drive multiple speakers to sound, so as to realize the virtual sound and image generation of the azimuth angle of the corresponding video image.

[0091] Specifically, the sound pressure difference △SPL is first converted into a gain and applied to the sound to be played. The gain applied to the sound to be played is determined by the change in the sound pressure level. Finally, a virtual sound image generation algorithm combined with the speaker layout is used to drive multiple speakers to produce sound, thereby realizing the generation of virtual sound images at the azimuth angle of the above-mentioned image.

[0092] It should be noted that the virtual sound image generation algorithm in this embodiment can be tangent, vector amplitude shifting, distance-based amplitude shifting, etc., which are not specifically limited here. Among them, the vector amplitude shifting technology uses the data of each specific speaker position and mixes the three speakers closest to the required position; this method is based on the directional component of the vector corresponding to the two or three speakers placed closest to the sound source. The distance-based amplitude shifting technology is based on amplitude shifting and is applicable to a series of speakers. The gain of each speaker is calculated by an attenuation model, which is established based on the distance information between the sound source position and each speaker.

[0093] To summarize, the method for realizing synchronous visual and auditory perception of the present application obtains the relative position and distance between the image and the audience and determines the type of room when the video is played, and dynamically adjusts the audio signal to be played according to the position and distance between the audience and the image and the type of room, generating a virtual sound and image that changes synchronously with the image, so that the audience can obtain the same spatial perception in hearing and vision, increasing the sense of immersion in video applications.

[0094] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0095] Reference Figure 4 , showing a structural block diagram of a device for realizing synchronous visual and auditory perception of the present application, the device 400 includes:

[0096] The first processing module 401 is configured to, when a video application is turned on, obtain relative position information between the played video image and the viewer and the type of room the viewer is in;

[0097] It should be noted that the first processing module 301 in this embodiment is a module with video acquisition, audience / speaker capture and distance calculation functions and room type detection functions. Here, it is unified as the first processing module, which can also be called a video acquisition module.

[0098] The second processing module 402 is configured to determine a volume adjustment gain based on the acquired relative position information and the type of room where the audience is located;

[0099] The third processing module 403 is configured to adjust the to-be-played sound of the video application according to the volume adjustment gain.

[0100] In some embodiments, the device for realizing synchronous visual and auditory perception of the present application is based on the principle of optical reversibility, and compares the video camera to the audience, and obtains the relative position information between the played video image and the audience through the video acquisition end.

[0101] In some embodiments, the device for realizing synchronous visual and auditory perception of the present application is based on the principle of parallax and uses multiple cameras arranged at the position of the audience to obtain the relative position information between the played video image and the audience.

[0102] In some embodiments, the device for achieving synchronous visual and auditory perception of the present application combines the orientation and angle of the viewer in front of the display screen with the imaging position and screen depth of the viewer on the display screen to determine the relative position information between the played video image and the viewer.

[0103] In some embodiments, the second processing module 402 in the apparatus for realizing synchronous visual and auditory perception of the present application is specifically configured as follows:

[0104] Determine a corresponding sound pressure attenuation function based on the type of room where the audience is located;

[0105] Substituting the distance value of the current video image frame and the distance value of the previous video image frame contained in the relative position information into the sound pressure attenuation function to calculate the sound pressure difference value;

[0106] When the sound pressure difference is greater than a preset threshold, the sound pressure difference is converted to obtain the volume adjustment gain.

[0107] In some embodiments, in the device for realizing synchronous visual and auditory perception of the present application, when the type of the room where the audience is located is a large space or soft decoration, the sound pressure attenuation function is a linear function;

[0108] When the room type where the audience is located is a long and narrow room, the sound pressure attenuation function is a logarithmic function;

[0109] When the room type where the audience is located is a small room, the sound pressure attenuation function is an inverse logarithmic function.

[0110] In some embodiments, the third processing module 403 in the device for realizing synchronous visual and auditory perception of the present application is specifically configured to: apply the volume adjustment gain to the sound to be played, and use a virtual sound and image generation algorithm to drive multiple speakers to sound, so as to realize the virtual sound and image generation of the azimuth angle of the corresponding video image.

[0111] It should be noted that, in this embodiment, the second processing module 402 and the third processing module 403 may also be combined into one, namely, a virtual sound image generation module.

[0112] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0113] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0114] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0115] Figure 5 is a block diagram of an electronic device 500 shown in the present application. For example, the electronic device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0116] Reference Figure 5 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0117] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0118] The memory 504 is configured to store various types of data to support operations on the device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, images, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0119] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0120] The multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0121] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0122] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0123] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0124] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0125] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0126] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0127] Figure 6 19 is a block diagram of a computer-readable storage medium 1900 shown in the present application. For example, the computer-readable storage medium 1900 may be provided as a server.

[0128] Reference Figure 6 , the computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0129] The computer readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer readable storage medium 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0130] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0132] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0133] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0135] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0138] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for achieving synchronous visual and auditory perception, characterized in that: The method comprises: When a video application is turned on, the relative position information between the played video image and the viewer and the type of room the viewer is in are obtained; Determining a volume adjustment gain based on the acquired relative position information and the type of room where the audience is located; The sound to be played of the video application is adjusted according to the volume adjustment gain.

2. A method for achieving synchronous visual and auditory perception according to claim 1, characterized in that: Based on the optical reversibility principle, the video camera is compared to an audience, and the relative position information between the played video image and the audience is obtained through the video acquisition end.

3. A method for achieving synchronous visual and auditory perception according to claim 1, characterized in that: Based on the parallax principle, multiple cameras arranged at the audience's location are used to obtain the relative position information between the played video image and the audience.

4. A method for achieving synchronous visual and auditory perception according to claim 1, characterized in that: The relative position information between the played video image and the viewer is determined by combining the viewer's position and angle in front of the display screen with the viewer's imaging position and screen-out depth on the display screen.

5. The method for realizing synchronous visual and auditory perception according to claim 1, characterized in that: The determining of the volume adjustment gain based on the acquired relative position information and the type of the room where the audience is located specifically includes: Determine a corresponding sound pressure attenuation function based on the type of room where the audience is located; Substituting the distance value of the current video image frame and the distance value of the previous video image frame contained in the relative position information into the sound pressure attenuation function to calculate the sound pressure difference value; When the sound pressure difference is greater than a preset threshold, the sound pressure difference is converted to obtain the volume adjustment gain.

6. A method for achieving synchronous visual and auditory perception according to claim 5, characterized in that: The corresponding sound pressure attenuation function is determined based on the type of room where the audience is located, specifically: When the room type where the audience is located is a large space or soft decoration, the sound pressure attenuation function is a linear function; When the room type where the audience is located is a long and narrow room, the sound pressure attenuation function is a logarithmic function; When the room type where the audience is located is a small room, the sound pressure attenuation function is an inverse logarithmic function.

7. The method for realizing synchronous visual and auditory perception according to claim 1, characterized in that: The adjusting of the sound to be played of the video application based on the volume adjustment gain is specifically: applying the volume adjustment gain to the sound to be played, and using a virtual sound and image generation algorithm to drive multiple speakers to sound, so as to achieve virtual sound and image generation at the azimuth angle of the corresponding video image.

8. A device for realizing synchronous visual and auditory perception, characterized in that: The device comprises: The first processing module is configured to, when the video application is turned on, obtain the relative position information between the played video image and the viewer and the type of room the viewer is in; A second processing module is configured to determine a volume adjustment gain based on the acquired relative position information and the type of room where the audience is located; The third processing module is configured to adjust the sound to be played of the video application according to the volume adjustment gain.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for realizing synchronous visual and auditory perception as claimed in any one of claims 1 to 7 is realized.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for realizing synchronous visual and auditory perception according to any one of claims 1 to 7 is implemented.

Images