Sound and picture co-location playing method and device, equipment and medium

Abstract

The invention relates to a sound and picture co-location playing method, device and equipment and a medium, and belongs to the technical field of audio and video signal processing, and the method comprises the steps: dividing a screen into a plurality of sound image regions from the center to the edge according to the length from short to long; acquiring first position information of a first sound source image in a screen; determining a first sound image area to which the first sound source image belongs in a screen according to the first position information; obtaining a first target orientation according to the first sound image area; and generating a virtual sound image of the first sound source image in the first target orientation, and performing sound playing through the virtual sound image. According to the scheme, the screen is divided into a limited number of fixed partition areas, the area in which the sound source image is displayed is judged, and the target generation direction of the virtual sound image is set as the geometric midpoint of the area, so that sound and picture co-location is realized.

Description

Technical Field

[0001] The present invention relates to the field of audio and video signal processing, and in particular to a method, device, equipment and medium for playing audio and video in the same position. Background Art

[0002] Immersive audio-visual technology has been widely used in the film and television industry. For example, in a cinema, virtual audio-visual positioning technology combined with 3D video can give audiences an immersive experience of a train whizzing by or a bomber flying overhead. However, in the fields of video conferencing, remote teaching, video presentations, etc., current products are far from achieving a sense of presence that satisfies users.

[0003] Some tricky methods are to display the image of a sound source on a fixed screen, which is bound to a single speaker near the screen. When it is confirmed that the sound source displayed on the screen is making the sound, the speaker corresponding to the screen will sound, giving the user in front of the screen a sense of sound and picture being in the same position. However, this method has a large deviation angle between sound and picture, and especially when the screen is large, the audience will still have the perception that the sound and picture are separated. Summary of the invention

[0004] In view of this, it is necessary to provide a method, device, equipment and medium for playing sound and picture in the same position, so as to solve the problem that the existing screen is bound to a single speaker near the screen. When it is confirmed that the sound source displayed on the screen makes a sound, the speaker corresponding to the screen makes a sound. In this method, the deviation angle of sound and picture is large, and the audience will have the problem of perceiving the separation of sound and picture.

[0005] In order to solve the above problems, the present invention provides a method for playing audio and video in the same position, comprising: Divide the screen from the center to the edge into multiple audio and video areas according to the length from short to long; Acquire first position information of a first sound source image on the screen; Determining, according to the first position information, a first sound image area to which the first sound source image belongs on the screen; Obtaining a first target orientation according to the first sound and image area; A virtual sound image of the first sound source image is generated at the first target orientation, and sound is played through the virtual sound image.

[0006] Optionally, after generating a virtual sound image of the first sound source image at the first target position and playing sound through the virtual sound image, the method further includes: The position information of the first sound source image is detected in real time, and when the first sound source image changes from a first position to a second position information belonging to the first sound image area, sound is played through the virtual sound image at the first target position.

[0007] Optionally, after generating a virtual sound image of the first sound source image at the first target position and playing sound through the virtual sound image, the method further includes: detecting the position information of the first sound source image in real time, when the first sound source image changes from the first position information to the third position information belonging to the second sound image area; Obtaining a second target orientation according to the second sound and image area; The virtual sound image is moved from the first target position to the second target position, and sound is played through the virtual sound image at the second target position.

[0008] Optionally, the real-time detection of the position information of the first sound source image specifically includes: Determine the detection frequency based on the audience's reaction time to the picture and the audio delay time; According to the detection frequency, position information of the first sound source image is detected.

[0009] Optionally, obtaining a first target orientation according to the first sound and image area specifically includes: A geometric midpoint of the first sound and image area is determined, and the geometric midpoint is determined as a first target orientation.

[0010] Optionally, dividing the screen into a plurality of audiovisual areas from the center to the edge according to the length from short to long specifically includes: The screen is divided into M horizontal areas and N vertical areas in a centrally symmetrical manner to obtain multiple sound and image areas; M and N are odd numbers respectively; the horizontal lengths of the multiple horizontal areas increase step by step from the center to the edge; the vertical lengths of the multiple vertical areas increase step by step from the center to the edge.

[0011] Optionally, the acquiring first position information of a first sound source image on the screen specifically includes: When the first sound source image is a human or an animal, the first position information is a key point of the mouth; When the first sound source image is an object, the first position information is the lateral geometric center of the object.

[0012] The present invention also provides a sound and image co-location playback device, comprising: a region division module, a sound source position acquisition module, a judgment module, a target orientation calculation module and a virtual sound and image generation and playback module; The area division module is used to divide the screen into a plurality of audio-visual areas from the center to the edge according to the length from short to long; The sound source position acquisition module is used to acquire first position information of a first sound source image on the screen; The determination module is used to determine the first sound image area to which the first sound source image belongs on the screen according to the first position information; The target orientation calculation module is used to obtain a first target orientation according to the first sound and image area; The virtual sound image generation and playback module is used to generate a virtual sound image of the first sound source image at the first target orientation, and to play sound through the virtual sound image.

[0013] Another technical solution of the present invention to solve the above technical problem is an electronic device, including a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of a method for playing audio and video in the same position in any of the above solutions.

[0014] Another technical solution of the present invention to solve the above technical problem is a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps of a method for playing audio and video in the same position as any of the above solutions.

[0015] By applying the technical solution provided in the embodiment of the present application, the screen is divided into a limited number of fixed partition areas, thereby reducing the hardware and algorithm overhead. By determining the sound and image area to which the sound source image belongs on the screen, the target position of the virtual sound image is determined, and the virtual sound image is played at the target position to achieve the sound and image synchronization effect of the sound source image, so as to adjust the position of the virtual sound image to the image position of the sound source on the screen, so that the audience perceives that the sound comes from the corresponding sound source image position on the screen, thereby achieving the sound and image synchronization effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of the steps of a method for playing audio and video in the same position provided in an embodiment of the present invention; Figure 2 is a schematic diagram of the coordinates of the sound source image and the corresponding virtual sound image target orientation provided in an embodiment of the present invention; Figure 3 is a flowchart of the steps of a method for playing audio and video in the same position provided in another embodiment of the present invention; Figure 4 is a flowchart of the steps of a method for playing audio and video in the same position provided in another embodiment of the present invention; Figure 5 is a flowchart of the steps of detecting the position information of the first sound source image provided in an embodiment of the present invention; Figure 6 is a schematic diagram of horizontal plane projection when M=5 and N=1 provided in an embodiment of the present invention; Figure 7 This is a flowchart of a method for audio and video synchronization provided in an embodiment of the present invention. Figure 8 It is a structural block diagram of a playback device for audio and video co-location provided in an embodiment of the present invention; Fig. 9 It is a hardware structure block diagram of an electronic device provided in each embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Reference Figure 1 As shown, a flowchart of a method for playing audio and video in the same position is shown, and the method can be applied to audio and video signal processing, such as Figure 1 As shown, the following steps may be specifically included: Step S101, dividing the screen into a plurality of audio-visual areas from the center to the edge according to the length from short to long; In this embodiment, step S101 may include: dividing the screen display area into a plurality of areas with dense centers and sparse edges in an uneven manner from short to long lengths.

[0019] For example, the screen is divided into M horizontal areas and N vertical areas in a centrally symmetrical manner to obtain multiple sound and image areas; M and N are limited to 10≥M≥3≥N≥1; M and N are odd numbers; the horizontal lengths of the multiple horizontal areas increase step by step from the center to the edge; the vertical lengths of the multiple vertical areas increase step by step from the center to the edge. The sound and image areas can be rectangular.

[0020] Step S102, obtaining first position information of a first sound source image on the screen; In this embodiment, refer to Figure 2 As shown, a schematic diagram of the sound source image coordinates and the corresponding virtual sound image target orientation is shown. The first sound source image may be an image of a little girl, and the first position information may be the center position of the mouth of the little girl image.

[0021] Step S103: determining, according to the first position information, a first sound image area to which the first sound source image belongs on the screen; In this embodiment, refer to Figure 2As shown, a schematic diagram of the coordinates of the sound source image and the corresponding virtual sound image target orientation is shown, and the screen is divided into five areas from left to right: virtual sound image area 1, virtual sound image area 2, virtual sound image area 3, virtual sound image area 4 and virtual sound image area 5. The center of the little girl's mouth is located in virtual sound image area 4, and the first sound image area is also virtual sound image area 4 in the figure.

[0022] Step S104: obtaining a first target orientation according to the first sound and image area; in this embodiment, referring to Figure 2 As shown, a schematic diagram of the sound source image coordinates and the corresponding virtual sound image target orientation is shown, and the geometric center of the virtual sound image area 4 can be determined as the first target orientation.

[0023] Step S105: Generate a virtual sound image of the first sound source image at the first target direction, and play sound through the virtual sound image.

[0024] In this embodiment, virtual sound image and virtual sound image positioning can be: after the sound emitted by multiple speakers is transmitted to the two ears, the listener will perceive the existence of some sound sources that are not in the actual position of the speakers. These sound sources themselves do not exist in the space where the listener is located, but are "imagined" by the listener's auditory system, so they are called virtual sound images, or phantom sound sources. By adjusting the output signals of multiple speakers using amplitude shifting, delay adjustment or hybrid methods, the listener can perceive the target position of the virtual sound image. The corresponding technologies include VBAP (vector-based amplitude shifting), DBAP (distance-based amplitude shifting), etc., which can also be called virtual sound image generation.

[0025] In this embodiment, virtual sound image generation can be performed at the target location using virtual sound image localization algorithms such as VBAP and DBAP according to the speaker configuration of the device where the screen is located. Figure 2 For example, the method for generating a virtual sound image may include: calculating different volumes of each speaker according to the angle between each speaker and the virtual sound image target orientation and the target listening point, and converting the volume into a gain to process the audio signal to be played.

[0026] In one scenario, the same location of sound and image can be: when playing a video on a screen, the position of the virtual sound and image is adjusted to the image position of the sound source on the screen through virtual sound and image positioning technology, so that the audience perceives that the sound comes from the corresponding sound source image position on the screen.

[0027] In one example, referring to Figure 3 As shown, a flowchart of a method for playing audio and video in the same position is shown, such as Figure 3 As shown, the following steps may be specifically included: Step S301, dividing the screen into a plurality of audio-visual areas from the center to the edge according to the length from short to long; For example, the screen is divided into M horizontal areas and N vertical areas in a centrally symmetrical manner to obtain multiple sound and image areas; M and N are limited to 10≥M≥3≥N≥1; M and N are odd numbers; the horizontal lengths of the multiple horizontal areas increase step by step from the center to the edge; the vertical lengths of the multiple vertical areas increase step by step from the center to the edge. The sound and image areas can be rectangular.

[0028] Step S302, obtaining first position information of a first sound source image on the screen; In this embodiment, refer to Figure 2 As shown, a schematic diagram of the sound source image coordinates and the corresponding virtual sound image target orientation is shown. The first sound source image may be an image of a little girl, and the first position information may be the center position of the mouth of the little girl image.

[0029] Step S303: determining, according to the first position information, a first sound image area to which the first sound source image belongs on the screen; In this embodiment, refer to Figure 2 As shown, a schematic diagram of the coordinates of the sound source image and the corresponding virtual sound image target orientation is shown, and the screen is divided into five areas from left to right: virtual sound image area 1, virtual sound image area 2, virtual sound image area 3, virtual sound image area 4 and virtual sound image area 5. The center of the little girl's mouth is located in virtual sound image area 4, and the first sound image area is also virtual sound image area 4 in the figure.

[0030] Step S304: obtaining a first target orientation according to the first sound and image area; in this embodiment, referring to Figure 2 As shown, a schematic diagram of the sound source image coordinates and the corresponding virtual sound image target orientation is shown, and the geometric center of the virtual sound image area 4 can be determined as the first target orientation.

[0031] Step S305: generating a virtual sound image of the first sound source image at the first target position, and playing sound through the virtual sound image; Step S306: detecting the position information of the first sound source image in real time, and when the first sound source image changes from the first position to the second position information belonging to the first sound image area, playing sound through the virtual sound image at the first target direction.

[0032] In this embodiment, the first sound source image changes from the first position to the second position information belonging to the first sound image area, which can be: Figure 2 As shown, the image of the little girl moves in the virtual sound and image area 4, but does not leave the virtual sound and image area 4, so the first target position remains unchanged.

[0033] In one example, referring to Figure 4 As shown, a flowchart of a method for playing audio and video in the same position is shown, such as Figure 4 As shown, the following steps may be specifically included: Step S401, dividing the screen into a plurality of audio-visual areas from the center to the edge according to the length from short to long; In this embodiment, step S401 may include: dividing the screen display area into a plurality of areas with dense centers and sparse edges in an uneven manner from short to long lengths.

[0034] For example, the screen is divided into M horizontal areas and N vertical areas in a centrally symmetrical manner to obtain multiple sound and image areas; M and N are limited to 10≥M≥3≥N≥1; M and N are odd numbers; the horizontal lengths of the multiple horizontal areas increase step by step from the center to the edge; the vertical lengths of the multiple vertical areas increase step by step from the center to the edge. The sound and image areas can be rectangular.

[0035] Step S402, obtaining first position information of a first sound source image on the screen; In this embodiment, refer to Figure 2 As shown, a schematic diagram of the sound source image coordinates and the corresponding virtual sound image target orientation is shown. The first sound source image may be an image of a little girl, and the first position information may be the center position of the mouth of the little girl image.

[0036] Step S403: determining, according to the first position information, a first sound image area to which the first sound source image belongs on the screen; In this embodiment, refer to Figure 2 As shown, a schematic diagram of the coordinates of the sound source image and the corresponding virtual sound image target orientation is shown, and the screen is divided into five areas from left to right: virtual sound image area 1, virtual sound image area 2, virtual sound image area 3, virtual sound image area 4 and virtual sound image area 5. The center of the little girl's mouth is located in virtual sound image area 4, and the first sound image area is also virtual sound image area 4 in the figure.

[0037] Step S404: Obtain a first target orientation according to the first sound and image area; in this embodiment, refer to Figure 2 As shown, a schematic diagram of the sound source image coordinates and the corresponding virtual sound image target orientation is shown, and the geometric center of the virtual sound image area 4 can be determined as the first target orientation.

[0038] Step S405: generating a virtual sound image of the first sound source image at the first target position, and playing sound through the virtual sound image; In this embodiment, virtual sound image generation can be performed at the target location using virtual sound image localization algorithms such as VBAP and DBAP according to the speaker configuration of the device where the screen is located. Figure 2 For example, the method for generating a virtual sound image may include: calculating different volumes of each speaker according to the angle between each speaker and the virtual sound image target orientation and the target listening point, and converting the volume into a gain to process the audio signal to be played.

[0039] Step S406, detecting the position information of the first sound source image in real time, when the first sound source image changes from the first position information to the third position information belonging to the second sound image area; Step S407, obtaining a second target orientation according to the second sound and image area; Step S408: Move the virtual sound image from the first target position to the second target position, and play sound through the virtual sound image at the second target position.

[0040] In this embodiment, the first sound source image changes from the first position information to the third position information belonging to the second sound image area, which can be: Figure 2 As shown, for example: the image of the little girl talking moves out of the virtual sound image area 4 and moves to the virtual sound image area 5, then the virtual sound image area 5 is the second sound image area. For another example: the sound unit of the little girl talking is transferred to the drum, and the horizontal size center of the drum is in the virtual sound image area 1, then the virtual sound image area 1 is the second sound image area, and the second sound source image is the drum.

[0041] In one example, referring to Figure 5 As shown, a flowchart of the steps of detecting the position information of the first sound source image is shown, such as Figure 5 As shown, the following steps may be specifically included: Step S501, determining the detection frequency according to the audience's reaction time to the picture and the audio delay time; Step S502: Detect the position information of the first sound source image according to the detection frequency.

[0042] In one scenario, according to relevant literature, ordinary viewers can perceive sound changes that lag behind the picture by 80ms, and the delay of the spatial audio renderer in the headphone field is often less than 60ms. This patent sets the video signal processing module to update the sound source image coordinates at a frequency greater than 20Hz, preferably 30Hz.

[0043] In one example, calculating the first target position may include the following steps: A geometric midpoint of the first sound and image area is determined, and the geometric midpoint is determined as a first target orientation.

[0044] In this embodiment, refer to Figure 2 As shown, a schematic diagram of the coordinates of the sound source image and the corresponding virtual sound image target orientation is shown. Specifically, the screen is divided into five areas from left to right: virtual sound image area 1, virtual sound image area 2, virtual sound image area 3, virtual sound image area 4 and virtual sound image area 5. The center of the mouth of the little girl's image is located in the virtual sound image area 4, and the first sound image area is the virtual sound image area 4 in the figure. The geometric center of the virtual sound image area 4 is the intersection of the diagonals of the rectangular virtual sound image area 4.

[0045] In one example, the screen is divided into M horizontal regions and N vertical regions in a centrosymmetric manner to obtain multiple sound and image regions; M and N are odd numbers; the horizontal lengths of the multiple horizontal regions increase step by step from the center to the edge; the vertical lengths of the multiple vertical regions increase step by step from the center to the edge. The sound and image regions may be rectangular.

[0046] In one scenario, the human auditory system has the highest accuracy in localizing sound images in the horizontal direction in front of it, and the localization accuracy decreases rapidly from the middle to the sides; related literature shows that when listening indoors to a stereo speaker system with a speaker spacing of D, the subject's perception accuracy of the virtual sound image in the middle of the horizontal dimension is about 7%*D, and the positioning becomes more blurred as it moves to the sides, and the localization accuracy is about 20%*D when it is close to the speaker. At the best listening position, the human auditory system can perceive 7-10 virtual sound images in the horizontal direction. The human auditory system is not sensitive to the movement of sound images in height, and sometimes the sound image movement will only be perceived when it produces an angular displacement of 60° at the subject.

[0047] For example, the screen display area is divided horizontally and vertically in a non-uniform manner from the center to both sides and from small to large, forming M horizontal and N vertical areas with dense centers and sparse edges, and M and N are limited to 10≥M≥3≥N≥1.

[0048] As another example, considering the algorithm complexity and the audience's perception accuracy, assuming that the screen has a horizontal width of L and the areas are numbered 1 to M from left to right, the width of the most central area d≤10%*L, that is: When M is an odd number, the width d of the (M+1) / 2th region is ≤ 10%*L; When M is an even number, the width d of the M / 2th and (M / 2+1)th regions is equal and ≤10%*L; The widths of other regions increase from the center to the sides in a trend of increasing step difference. Preferably, in order to form the difference in sound image position between the center and the sides, M and N are odd numbers.

[0049] In one example, referring to Figure 6As shown, a schematic diagram of horizontal plane projection is shown, such as Figure 6 As shown, when M=5 and N=1, the horizontal width of each area accounts for 30%, 15%, 10%, 15%, and 30% of the screen width.

[0050] In an example, obtaining the first location information may specifically include the following steps: When the first sound source image is a human or an animal, the first position information is a key point of the mouth; When the first sound source image is an object, the first position information is the lateral geometric center of the object.

[0051] In one example, referring to Figure 7 As shown, a flow chart of the steps of a method for audio and video co-location is shown, such as Figure 7 As shown, the following steps may be specifically included: dividing the screen from the center to the edge into a plurality of regions non-uniformly from short to long; establishing a coordinate system in the screen display region; Providing the initial coordinates of the sound source image on the screen; in this embodiment, a coordinate system is established for the display area of ​​the screen, and the above partitions are mapped to the display coordinate system of the screen to confirm the coordinate range of each area, and confirm the coordinates of the geometric center within the area; When the screen displays a video image, the video signal processing module gives the coordinate point of the sound source.

[0052] For the image of the speaker, give the coordinates of the center point of the mouth; For non-human sound sources, give the coordinates of the midpoint of the horizontal dimension of its image.

[0053] For example, a Cartesian coordinate system may be established with the lower left corner of the screen as the origin.

[0054] Determine in which region the sound source image is located, and take the geometric midpoint of the current region as the target orientation; Generate virtual sound and image; In this embodiment, virtual sound and image localization algorithms such as VBAP and DBAP can be used according to the speaker configuration of the device where the screen is located to generate virtual sound and image at the target location. Figure 2 For example, the method for generating a virtual sound image may include: calculating different volumes of each speaker according to the angle between each speaker and the virtual sound image target orientation and the target listening point, and converting the volume into a gain to process the audio signal to be played.

[0055] Refresh the coordinates of the sound source image on the screen; Determine whether the sound source image leaves the previous area. If so, take the geometric midpoint of the new area as the target orientation and refresh the coordinates of the sound source image on the screen.

[0056] If not, the geometric midpoint of the current area is used as the target orientation, virtual sound image is generated, and the coordinates of the sound source image on the screen are refreshed.

[0057] In this embodiment, the video signal processing module dynamically updates the coordinates of the sound source image at a fixed frequency, and the audio signal processing module determines whether to update the target generation direction of the virtual sound image according to the coordinates.

[0058] When the sound source image on the screen is displaced significantly, that is, it leaves the current mapping area, sound and image separation will occur. At this time, the audio signal processing module will regenerate the virtual sound image according to the new sound source image coordinates; otherwise, the virtual sound image position does not need to be adjusted.

[0059] Specifically, according to relevant literature, ordinary viewers can perceive sound changes that lag behind the picture by 80ms, and the delay of the spatial audio renderer in the headphone field is often less than 60ms. This patent sets the video signal processing module to update the sound source image coordinates at a frequency greater than 20Hz, preferably 30Hz.

[0060] For example, when the embodiment of the present application is applied to the playback of a speech video, the screen display area is non-uniformly divided into a number of rectangular areas with dense middles and sparse edges according to the length from short to long, and the area in which the speaker's sound source image on the screen is located is dynamically identified and confirmed. Then, through virtual sound image positioning technology, a virtual sound image is generated with the geometric center of the area as the target position, and the output signals of multiple speakers are adjusted by amplitude shifting, time delay adjustment or a mixed method, so that people watching the speech can perceive the target position of the virtual sound image.

[0061] When the embodiment of the present application is applied to video conference playback, the screen display area is non-uniformly divided into a number of rectangular areas with dense middles and sparse edges according to length from short to long, and the area in which the voice source image of the conference person on the screen is located is dynamically identified and confirmed. Then, through virtual sound and image positioning technology, a virtual sound and image is generated with the geometric center of the area as the target position, and the output signals of multiple speakers are adjusted using amplitude shifting, delay adjustment or a mixed method, so that people watching the conference can perceive the target position of the virtual sound and image.

[0062] When the embodiment of the present application is applied to live broadcast, the screen display area is non-uniformly divided into a number of rectangular areas with dense middles and sparse edges according to the length from short to long, and the area where the live human voice source image on the screen is located is dynamically identified and confirmed. Then, through virtual sound and image positioning technology, a virtual sound and image is generated with the geometric center of the area as the target position, and the output signals of multiple speakers are adjusted by amplitude shifting, delay adjustment or mixed methods, so that people watching the live broadcast can perceive the target position of the virtual sound and image.

[0063] When the embodiment of the present application is applied to remote teaching playback, the screen display area is non-uniformly divided into a number of rectangular areas with dense middles and sparse edges according to length from short to long, and the area where the sound source image of the lecturer on the screen is located is dynamically identified and confirmed. Then, through virtual sound image positioning technology, a virtual sound image is generated with the geometric center of the area as the target position, and the output signals of multiple speakers are adjusted by amplitude shifting, time delay adjustment or mixed methods, so that the lecturer can perceive the target position of the virtual sound image.

[0064] In one example, referring to Figure 8 As shown, a structural block diagram of a playback device for audio and video co-location according to an embodiment of the present invention is shown. Figure 8 As shown, the device may specifically include the following modules: The area division module 801 is used to divide the screen into a plurality of audio-visual areas from the center to the edge according to the length from short to long; A sound source position acquisition module 802 is used to acquire first position information of a first sound source image on the screen; A determination module 803 is used to determine the first sound image area to which the first sound source image belongs on the screen according to the first position information; A target orientation calculation module 804, configured to obtain a first target orientation according to the first sound and image area; The virtual sound image generation and playback module 805 is used to generate a virtual sound image of the first sound source image at the first target position, and play sound through the virtual sound image.

[0065] In an example, the device may further include the following modules: The real-time monitoring module is used to detect the position information of the first sound source image in real time, and when the first sound source image changes from a first position to a second position information belonging to the first sound image area, the sound is played through the virtual sound image at the first target direction.

[0066] In an example, the device may further include the following modules: The real-time monitoring module is further used to detect the position information of the first sound source image in real time, when the first sound source image changes from the first position information to the third position information belonging to the second sound image area; Obtaining a second target orientation according to the second sound and image area; The virtual sound image is moved from the first target position to the second target position, and sound is played through the virtual sound image at the second target position.

[0067] In one example, the real-time monitoring module is further specifically used to determine the detection frequency according to the audience's reaction time to the picture and the audio delay time; According to the detection frequency, position information of the first sound source image is detected.

[0068] In one example, the target orientation calculation module is specifically configured to determine a geometric midpoint of the first sound and image area, and determine the geometric midpoint as the first target orientation.

[0069] In one example, the area division module is specifically used to divide the screen into M horizontal areas and N vertical areas in a centrally symmetrical manner to obtain multiple sound and image areas; M and N are odd numbers respectively; the horizontal lengths of the multiple horizontal areas increase step by step from the center to the edge; the vertical lengths of the multiple vertical areas increase step by step from the center to the edge.

[0070] In one example, the sound source position acquisition module is specifically used for when the first sound source image is a human or an animal, the first position information is a mouth key point; When the first sound source image is an object, the first position information is the lateral geometric center of the object.

[0071] The above embodiment provides a device for playing audio and video in the same position, which can realize the technical solution described in the above embodiment of a method for playing audio and video in the same position. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of a method for playing audio and video in the same position, which will not be repeated here.

[0072] In one example, referring to Fig. 9 As shown, a structural block diagram of an electronic device 900 according to an embodiment of the present invention is shown. Fig. 9 As shown, the electronic device 900 includes a processor 901 , a memory 902 and a display 903 . Fig. 9 Only some components of the electronic device 900 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0073] In some embodiments, the memory 902 may be an internal storage unit of the electronic device 900, such as a hard disk or memory of the electronic device 900. In other embodiments, the memory 902 may also be an external storage device of the electronic device 900, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device 900.

[0074] Furthermore, the memory 902 may include both an internal storage unit of the electronic device 900 and an external storage device. The memory 902 is used to store application software installed in the electronic device 900 and various data.

[0075] In some embodiments, the processor 901 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 902, such as a method for playing audio and video in the same position in the present invention.

[0076] In some embodiments, the display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 903 is used to display information on the electronic device 900 and to display a visual user interface. The components 901-903 of the electronic device 900 communicate with each other via a system bus.

[0077] In some embodiments of the present invention, when the processor 901 executes the audio and video co-location playback program in the memory 902, the following steps may be implemented: Divide the screen into a plurality of sound and image areas from the center to the edge according to the length from short to long; obtain first position information of a first sound source image in the screen; determine the first sound and image area to which the first sound source image belongs in the screen according to the first position information; obtain a first target orientation according to the first sound and image area; A virtual sound image of the first sound source image is generated at the first target orientation, and sound is played through the virtual sound image.

[0078] It should be understood that: when the processor 901 executes the audio and video co-location playback program in the memory 902, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0079] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 900 mentioned, and the electronic device 900 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 900 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0080] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute a method for playing sound and image in the same position as provided by the above-mentioned methods, the method comprising: dividing the screen into a plurality of sound and image areas from the center to the edge according to lengths from short to long; obtaining first position information of a first sound source image in the screen; determining, based on the first position information, a first sound and image area to which the first sound source image belongs in the screen; obtaining a first target orientation based on the first sound and image area; generating a virtual sound and image of the first sound source image at the first target orientation, and playing sound through the virtual sound and image.

[0081] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0082] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0083] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0084] The above is a detailed introduction to the audio and video co-location playback method, device, equipment and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for playing audio and video in the same position, characterized in that: include: Divide the screen from the center to the edge into multiple audio and video areas according to the length from short to long; Acquire first position information of a first sound source image on the screen; Determining, according to the first position information, a first sound image area to which the first sound source image belongs on the screen; Obtaining a first target orientation according to the first sound and image area; A virtual sound image of the first sound source image is generated at the first target orientation, and sound is played through the virtual sound image.

2. The method for playing audio and video in the same position as claimed in claim 1, characterized in that: After generating a virtual sound image of the first sound source image at the first target position and playing sound through the virtual sound image, the method further includes: The position information of the first sound source image is detected in real time, and when the first sound source image changes from a first position to a second position information belonging to the first sound image area, sound is played through the virtual sound image at the first target position.

3. A method for playing audio and video in the same position as claimed in claim 1 or 2, characterized in that: After generating a virtual sound image of the first sound source image at the first target position and playing sound through the virtual sound image, the method further includes: detecting the position information of the first sound source image in real time, when the first sound source image changes from the first position information to the third position information belonging to the second sound image area; Obtaining a second target orientation according to the second sound and image area; The virtual sound image is moved from the first target position to the second target position, and sound is played through the virtual sound image at the second target position.

4. A method for playing audio and video in the same position as claimed in claim 3, characterized in that: The real-time detection of the position information of the first sound source image specifically includes: Determine the detection frequency based on the audience's reaction time to the picture and the audio delay time; According to the detection frequency, position information of the first sound source image is detected.

5. A method for playing audio and video in the same position as claimed in claim 1 or 2, characterized in that: The obtaining of the first target position according to the first sound and image area specifically includes: A geometric midpoint of the first sound and image area is determined, and the geometric midpoint is determined as a first target orientation.

6. A method for playing audio and video in the same position as claimed in claim 1 or 2, characterized in that: The method of dividing the screen from the center to the edge into a plurality of audiovisual areas according to the length from short to long specifically includes: The screen is divided into M horizontal areas and N vertical areas in a centrally symmetrical manner to obtain multiple sound and image areas; M and N are odd numbers respectively; the horizontal lengths of the multiple horizontal areas increase step by step from the center to the edge; the vertical lengths of the multiple vertical areas increase step by step from the center to the edge.

7. A method for playing audio and video in the same position as claimed in claim 1 or 2, characterized in that: The obtaining of first position information of a first sound source image on the screen specifically includes: When the first sound source image is a human or an animal, the first position information is a key point of the mouth; When the first sound source image is an object, the first position information is the lateral geometric center of the object.

8. A playback device for audio and video co-location, characterized in that: include: Area division module, sound source location acquisition module, judgment module, target orientation calculation module and virtual sound and image generation and playback module; The area division module is used to divide the screen into a plurality of audio-visual areas from the center to the edge according to the length from short to long; The sound source position acquisition module is used to acquire first position information of a first sound source image on the screen; The determination module is used to determine the first sound image area to which the first sound source image belongs on the screen according to the first position information; The target orientation calculation module is used to obtain a first target orientation according to the first sound and image area; The virtual sound image generation and playback module is used to generate a virtual sound image of the first sound source image at the first target orientation, and to play sound through the virtual sound image.

9. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the audio and video co-location playback method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the audio and video co-location playback method described in any one of claims 1 to 7.

Images