Sound and picture co-location playing method, device and equipment

Abstract

The invention relates to a sound and picture co-location playing method, device and equipment, and relates to the field of audio and video signal processing. The method comprises the steps of obtaining an angle region mapping relation according to face orientation information of audiences; the angle area mapping relation is a mapping relation between the screen and the audience view area; acquiring first position information of a first sound source image in a screen; obtaining a screen area according to the first position information; obtaining an angle region corresponding to the screen region according to the angle region mapping relation and the screen region, and determining the angle region as a first sound image region; obtaining a first target orientation according to the first sound image area; and generating a virtual sound image of the first sound source image in the first target orientation, and performing sound playing through the virtual sound image. The hardware and algorithm overhead is effectively saved by processing a limited number of angle areas, and the audience can generate enough sound and picture co-location feeling by generating the virtual sound image in the target orientation, so that immersive audio and video experience is brought.

Description

Technical Field

[0001] The present invention relates to the field of audio and video signal processing, and in particular to a method, device and equipment for playing audio and video in the same position. Background Art

[0002] Immersive audio-visual technology has been widely used in the film and television industry. For example, in a cinema, virtual audio-visual positioning technology combined with 3D video can give audiences an immersive experience of a train whizzing by or a bomber flying overhead. However, in the fields of video conferencing, remote teaching, video presentations, etc., current products are far from achieving a sense of presence that satisfies users.

[0003] Some tricky methods are to display the image of a sound source on a fixed screen, which is bound to a single speaker near the screen. When it is confirmed that the sound source displayed on the screen is making the sound, the speaker corresponding to the screen will sound, giving the user in front of the screen a sense of sound and picture being in the same position. However, this method has a large deviation angle between sound and picture, and especially when the screen is large, the audience will still have the perception that the sound and picture are separated. Summary of the invention

[0004] In view of this, it is necessary to provide a method, device and equipment for playing sound and picture in the same position, so as to solve the problem that the existing screen is bound to a single speaker near the screen. When it is confirmed that the sound source displayed on the screen makes a sound, the speaker corresponding to the screen makes a sound. In this method, the deviation angle of sound and picture is large, and the audience will have the problem of perceiving the separation of sound and picture.

[0005] In order to solve the above problems, the present invention provides a method for playing audio and video in the same position, comprising: According to the facial orientation information of the audience, an angle-area mapping relationship is obtained; the angle-area mapping relationship is a mapping relationship between the screen and the audience's field of view; Acquire first position information of a first sound source image on the screen; Obtaining the screen area according to the first position information; According to the angle-area mapping relationship and the screen area, an angle area corresponding to the screen area is obtained, and the angle area is determined as a first sound-image area; Obtaining a first target orientation according to the first sound and image area; A virtual sound image of the first sound source image is generated at the first target orientation, and sound is played through the virtual sound image.

[0006] Optionally, obtaining the angle-area mapping relationship according to the facial orientation information of the audience specifically includes: Detect the audience's facial orientation in real time and obtain the audience's facial orientation information; Acquiring vertical distance data between the viewer and the screen according to the facial orientation information; According to the vertical data, the audience's visual field area is symmetrically divided into a plurality of angle areas from small to large angles; An angle region mapping relationship between the screen and a plurality of angle regions is established.

[0007] Optionally, dividing the audience's field of view area into a plurality of angle areas symmetrically from small to large angles according to the vertical data specifically includes: The audience's visual field is divided into M horizontal sector-shaped areas and N vertical sector-shaped areas in a centrally symmetrical manner; wherein M and N are odd numbers respectively; The angle of the fan-shaped area close to the center is no more than 5 degrees, and the angles of other fan-shaped areas increase gradually from the center to the edge; the audience field of view is a fan-shaped area with a centrally symmetrical angle of 60 degrees.

[0008] Optionally, obtaining a first target orientation according to the first sound and image area specifically includes: When the first sound source image is within the audience's field of view, the intersection of the angle bisector of the first mapping area on the screen is obtained, and the intersection is determined as the first target orientation.

[0009] Optionally, obtaining a first target orientation according to the first sound and image area specifically includes: When the first sound source image is outside the audience's field of view, the intersection of the boundary line of the outermost sector of the audience's field of view and the screen is obtained, and the intersection is determined as the first target orientation.

[0010] Optionally, obtaining a first target orientation according to the first sound and image area specifically includes: When the first sound source image is in the outermost sector of the audience's field of view, and the angle bisectors of the first mapping area have no intersection on the screen, the midpoint of the boundary of the outermost sector on the screen is determined as the first target orientation.

[0011] Optionally, after generating a virtual sound image of the first sound source image at the first target position and playing sound through the virtual sound image, the method further includes: Detecting the position information of the first sound source image in real time; When the first sound source image changes from the first position information to the second position information belonging to the first sound image area, sound is played through the virtual sound image at the first target position.

[0012] Optionally, the generating of a virtual sound image of the first sound source image at the first target position and after playing sound through the virtual sound image further comprises: Detecting the position information of the first sound source image in real time; When the first sound source image changes from the first position information to the third position information belonging to the second sound image area, obtaining a second target orientation according to the second sound image area; The virtual sound image is moved from the first target position to the second target position, and sound is played through the virtual sound image at the second target position.

[0013] The present invention also provides a playback device for co-location of sound and image, comprising: a mapping relationship building module, a position information acquisition module, a mapping area acquisition module, a target position information acquisition module and a virtual sound and image generation module; The mapping relationship building module is used to obtain an angle-area mapping relationship according to the facial orientation information of the audience; the angle-area mapping relationship is a mapping relationship between the screen and the audience's visual field area; the position information acquisition module is used to acquire the first position information of the first sound source image on the screen; The mapping area acquisition module is used to obtain the screen area according to the first position information; then obtain the angle area corresponding to the screen area according to the angle area mapping relationship and the screen area, and determine the angle area as the first sound and image area; The target position information acquisition module is used to obtain the first target position according to the first sound and image area; The virtual sound image generation module is used to generate a virtual sound image of the first sound source image at the first target orientation, and to play sound through the virtual sound image.

[0014] Another technical solution of the present invention to solve the above technical problem is an electronic device, including a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of a method for playing audio and video in the same position in any of the above solutions.

[0015] By applying the technical solution provided in the embodiments of the present application, the central area of ​​the audience's front field of vision is unevenly divided into a number of sector-shaped areas with dense centers and sparse edges in the horizontal and vertical directions by monitoring the audience's facial orientation and the vertical distance from the screen; the size of each sector is mapped on the screen; it is dynamically confirmed that the sound source image on the screen is located in which two sectors of the mapped intersection area; and then a virtual sound and image is generated at the target position. By processing a limited number of angular areas, the hardware and algorithm overhead is effectively saved. By generating a virtual sound and image at the target position, the audience can have a sufficient sense of sound and image co-location, bringing an immersive audio and video experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1It is a flowchart of the steps of a method for playing audio and video in the same position provided in an embodiment of the present invention; Figure 2 The present invention Figure 1 A flowchart of an embodiment of step S103; Figure 3 This is an example diagram of the division of the audience's front viewing angle when M=5 and N=1 provided in an embodiment of the present invention; Figure 4 It is a schematic diagram of a virtual sound image generation strategy provided in an embodiment of the present invention when the sound source image is outside the mapping area or the angle bisector plane is outside the screen; Figure 5 is a flowchart of the steps of a method for playing audio and video in the same position provided in another embodiment of the present invention; Figure 6 is a flowchart of the steps of a method for playing audio and video in the same position provided in another embodiment of the present invention; Figure 7 is a flow chart of the steps of a method for audio and video co-location provided in an embodiment of the present invention; Figure 8 It is a schematic diagram of the mapping of the included angle on the screen and the determination of the virtual sound and image position provided in the embodiment of the present invention; Fig. 9 It is a structural block diagram of a playback device for audio and video co-location provided in an embodiment of the present invention; Fig.10 It is a hardware structure block diagram of an electronic device provided in each embodiment of the present invention. DETAILED DESCRIPTION

[0017] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0018] like Figure 1 As shown, a specific embodiment of the present invention discloses a method for playing audio and video in the same position, comprising: S101. Obtain an angle region mapping relationship according to the facial orientation information of the audience; the angle region mapping relationship is a mapping relationship between the screen and the audience's field of view; it should be noted that, in an optional embodiment, the facial orientation of the audience is monitored in real time, and the central area of ​​the audience's front field of view is divided into angles in a non-uniform manner from the center to both sides, from small to large, and in a non-uniform manner; M horizontal and N vertical sector areas with dense centers and sparse edges are formed, and the limits of M and N are 10≥M≥3≥N≥1, M and N are odd numbers, and a mapping relationship is established between each sector area and the square area on the screen corresponding to the sector area, which is the angle region mapping relationship. For example: the central area of ​​the audience's front field of view is the i-th sector area horizontally and the i-th sector area vertically, and the i-th sector area horizontally and the i-th sector area vertically are mapped on the screen, and a long horizontal or vertical i-th rectangular plane area is obtained on the screen; each sector in the central area of ​​the audience's front field of view is mapped to the screen. i≥1.

[0019] S102, obtaining first position information of a first sound source image on the screen; S103, obtaining a screen area according to the first position information; S104, according to the angle area mapping relationship and the screen area, obtain the angle area corresponding to the screen area, and determine the angle area as the first sound and image area; it should be noted that, in an optional embodiment, for example, image A is at point a on the screen, point a is located in the i-th rectangular plane area of ​​the screen, and the i-th sector area (which may include horizontal and vertical sector areas) has a mapping relationship with the i-th rectangular plane area and the audience's central field of view area. That is, point a represents the first position information of image A, the i-th rectangular plane area is the screen area; the i-th sector area is the first sound and image area.

[0020] S105. Obtain a first target orientation according to the first audio-visual area. It should be noted that, in an optional embodiment, the intersection of the angle bisector of the i-th sector area on the screen is used as the first target orientation.

[0021] S106: Generate a virtual sound image of the first sound source image at the first target direction, and play sound through the virtual sound image.

[0022] It should be noted that, in a specific embodiment, virtual sound image and virtual sound image positioning can be: after the sound emitted by multiple speakers is transmitted to the two ears, the listener will perceive the existence of some sound sources that are not in the actual position of the speakers. These sound sources themselves do not exist in the space where the listener is located, but are "imagined" by the listener's auditory system, so they are called virtual sound images, or phantom sound sources. By adjusting the output signals of multiple speakers by amplitude shifting, delay adjustment or hybrid methods, the listener can perceive the target position of the virtual sound image. The corresponding technologies include VBAP (vector-based amplitude shifting), DBAP (distance-based amplitude shifting), etc., which can also be called virtual sound image generation. The method for generating virtual sound images can include: according to the angle between each speaker and the target position of the virtual sound image to the target listening point, to calculate the different volumes of each speaker, and convert it into gain and process the audio signal to be played at the target position.

[0023] In a specific embodiment, the sound and image are co-located: when a video is played on a screen, the position of the virtual sound and image is adjusted to the image position of the sound source on the screen through virtual sound and image positioning technology, so that the audience perceives that the sound comes from the corresponding sound source image position on the screen.

[0024] The solution of the present invention is mainly applied to one-to-one video scenes, taking into account the situation where there is only one audience member, and then obtaining facial orientation information in real time for processing according to the facial orientation of the audience member.

[0025] When the embodiment of the present application is applied to a video conference, the facial orientation of the conference participants is monitored in real time, and the central area of ​​the conference participants' front visual field is divided into angles in a non-uniform manner from the center to both sides, from small to large, and in a non-uniform manner; M horizontal and N vertical sector areas are formed, which are dense in the middle and sparse at the edges, and a mapping relationship is established between each sector area and the square area on the screen corresponding to the sector area, which is the angle area mapping relationship. When the facial orientation of the conference participants changes, the angle area mapping relationship will also change accordingly. For example, with the 60° sector area in the center of the conference participants' front visual field, a mapping relationship between each sector area and the square area on the screen corresponding to the sector area is established. If the conference participant's face is facing left, a 60° sector area is established in the left front, and if the conference participant's face is facing right, a 60° sector area is established in the right front. Based on the determined 60° sector area, a mapping relationship between each sector area and the square area on the screen corresponding to the sector area is established to obtain the corresponding angle area mapping relationship.

[0026] Dynamically confirm which two sectors' mapped intersection area the sound source image on the screen is located in; then generate a virtual sound image through virtual sound image localization technology, taking the intersection of the two sector angle bisectors on the screen as the target orientation.

[0027] When the embodiment of the present application is applied to the playback of a speech video, the facial orientation of the person watching the speech is monitored in real time, and the central area of ​​the front field of the person watching the speech is divided into angles in a non-uniform manner from the center to both sides, from small to large, and in a non-uniform manner; M horizontal and N vertical sector areas are formed, which are dense in the middle and sparse at the edges, and a mapping relationship is established between each sector area and the square area on the screen corresponding to the sector area, which is the angle area mapping relationship. When the facial orientation of the person watching the speech changes, the angle area mapping relationship will also change accordingly. For example, with a 60° sector area in the center of the front field of the person watching the speech, a mapping relationship is established between each sector area and the square area on the screen corresponding to the sector area. If the face of the person watching the speech faces left, a 60° sector area is established in the left front, and if the face of the person watching the speech faces right, a 60° sector area is established in the right front. Based on the determined 60° sector area, a mapping relationship is established between each sector area and the square area on the screen corresponding to the sector area, and the corresponding angle area mapping relationship is obtained.

[0028] Dynamically confirm which two sectors' mapping intersection area the speaker's image on the screen is located in; then generate virtual sound and image through virtual sound and image localization technology, taking the intersection of the two sectors' angular bisectors on the screen as the target orientation.

[0029] When the embodiment of the present application is applied to live broadcast, the facial orientation of the live broadcast viewer is monitored in real time, and the central area of ​​the front field of the live broadcast viewer is divided into angles in a non-uniform manner from the center to both sides, from small to large, and in a non-uniform manner; M horizontal and N vertical sector areas with dense middle and sparse edges are formed, and a mapping relationship is established between each sector area and the square area on the screen corresponding to the sector area, which is the angle area mapping relationship. When the facial orientation of the live broadcast viewer changes, the angle area mapping relationship will also change accordingly. For example, with the 60° sector area in the center of the front field of the live broadcast viewer, a mapping relationship between each sector area and the square area on the screen corresponding to the sector area is established. If the face of the live broadcast viewer faces left, a 60° sector area is established in the left front, and if the face of the live broadcast viewer faces right, a 60° sector area is established in the right front. Based on the determined 60° sector area, a mapping relationship between each sector area and the square area on the screen corresponding to the sector area is established to obtain the corresponding angle area mapping relationship.

[0030] Dynamically confirm which two sectors' mapped intersection area the live broadcast person's image on the screen is located in; then use virtual sound and image positioning technology to generate virtual sound and image with the intersection of the two sector angle bisecting planes on the screen as the target orientation.

[0031] When the embodiment of the present application is applied to remote teaching, the facial orientation of the students is monitored in real time, and the central area of ​​the student's front visual field is divided into angles in a non-uniform manner from the center to both sides, from small to large, and in a non-uniform manner; M horizontal and N vertical sector areas are formed, which are dense in the middle and sparse at the edges, and a mapping relationship is established between each sector area and the square area on the screen corresponding to the sector area, which is the angle area mapping relationship. When the student's facial orientation changes, the angle area mapping relationship will also change accordingly. For example, with the 60° sector area in the center of the student's front visual field, a mapping relationship between each sector area and the square area on the screen corresponding to the sector area is established. If the student's face is facing left, a 60° sector area is established in the left front, and if the student's face is facing right, a 60° sector area is established in the right front. Based on the determined 60° sector area, a mapping relationship between each sector area and the square area on the screen corresponding to the sector area is established to obtain the corresponding angle area mapping relationship.

[0032] Dynamically confirm the mapping intersection area of ​​the two sectors where the image of the teacher on the screen is located; then generate virtual sound and image through virtual sound image positioning technology, using the intersection of the two sector angle bisectors on the screen as the target orientation. By monitoring the audience's facial orientation and the vertical distance from the screen, the central area of ​​the audience's front field of view is unevenly divided into several sector areas with dense middle and sparse edges in the horizontal and vertical directions; the size of each sector is mapped on the screen; dynamically confirm the mapping intersection area of ​​the two sectors where the sound source image on the screen is located; then generate virtual sound and image through virtual sound image positioning technology, using the intersection of the two sector angle bisectors on the screen as the target orientation, so that the audience can have enough sound and image co-location sense with limited hardware and algorithm overhead, bringing an immersive audio and video experience. Among them, the mapping intersection area of ​​the two sectors represents the intersection area of ​​the horizontal sector and the vertical sector on the screen. Because the mapping of the sector on the screen is a long horizontal or vertical rectangular plane, the mapping intersection area of ​​the two sectors is a rectangular grid.

[0033] In an optional embodiment of the present invention, Figure 2 As shown, the angle-area mapping relationship is obtained according to the facial orientation information of the audience, specifically including: S201, detecting the facial orientation of the audience in real time, and obtaining the facial orientation information of the audience; S202, acquiring vertical distance data between the viewer and the screen according to the facial orientation information; S203, dividing the audience's field of view into a plurality of angle areas symmetrically from small to large angles according to the vertical data; S204: Establish an angle region mapping relationship between the screen and a plurality of angle regions.

[0034] By detecting the audience's facial orientation in real time to determine the mapping relationship between the audience's field of view and the screen area, it is possible to maintain a good sense of sound and picture co-location after the audience adjusts the viewing angle, thereby improving the audio and video experience.

[0035] In an optional embodiment of the present invention, the detection frequency is determined based on the audience's reaction time to the picture combined with the audio delay time; The facial orientation of the audience is detected in real time according to the detection frequency to obtain the facial orientation information of the audience.

[0036] It protects the use of a detection frequency greater than 20 Hz to detect changes in the audience's facial orientation. Since ordinary viewers can perceive sound changes that lag behind the picture by 80ms, the delay of the spatial audio renderer in the headphone field is often less than 60ms. The detection frequency greater than 20 Hz not only ensures the audience's viewing experience without noticing the separation of sound and picture, but also saves computing power.

[0037] In a certain embodiment, establishing the angle area mapping relationship between the screen and the multiple angle areas may specifically include: establishing a mapping relationship between each sector area and the square area on the screen corresponding to the sector area, that is, the angle area mapping relationship. For example: the i-th sector area in the horizontal direction and the i-th sector area in the vertical direction in the central area of ​​the viewer's front field of vision, the i-th sector area in the horizontal direction and the i-th sector area in the vertical direction are mapped on the screen, and a long horizontal or vertical i-th rectangular plane area is obtained on the screen; each sector in the central area of ​​the viewer's front field of vision is mapped to the screen. i≥1.

[0038] In an optional embodiment of the present invention, the step of symmetrically dividing the audience's field of view into a plurality of angle areas from small to large angles according to the vertical data specifically includes: The audience's visual field is divided into M horizontal sector areas and N vertical sector areas in a centrally symmetrical manner; wherein M and N are odd numbers, respectively, for forming a difference in the sound and image position in the middle and on both sides; in one embodiment, as Figure 3 As shown, when M=5 and N=1, the angles of each sector are 25°, 15°, 5°, 15°, and 25° respectively.

[0039] The angle of the fan-shaped area close to the center is no more than 5 degrees, and the angles of other fan-shaped areas increase gradually from the center to the edge; the audience field of view is a fan-shaped area with a centrally symmetrical angle of 60 degrees.

[0040] It should be noted that the human auditory system has the highest accuracy in the horizontal direction of the sound image in front, and the positioning accuracy decreases rapidly from the middle to the sides; related literature shows that the average accuracy of the subjects' perception of the virtual sound image at the horizontal angle in front is about 4°, and the positioning becomes more blurred as it moves to the sides. The human auditory system is not sensitive to the movement of sound images at height, and sometimes the sound image movement will not be perceived until it has an angular displacement of 60° at the subject.

[0041] In a certain embodiment, considering the complexity of the algorithm and the perception accuracy of the audience, when the sector areas are numbered from left to right as 1 to M, the angle of the most central sector is ≤5°, that is: When M is an odd number, the angle of the (M+1) / 2th sector is ≤5°; When M is an even number, the angle between the M / 2 and (M / 2+1) sectors is ≤5°; The angles of other sectors increase in a step-by-step manner from the center to the sides.

[0042] Because the angle between the speaker near the screen and the audience, that is, the listening angle, is smaller than the viewing angle range, this patent only divides the fan-shaped area of ​​about 60° in the center of the viewing angle in front of the audience, and ignores the edge of the viewing angle. Therefore, the sum of the angles of each fan-shaped area in the horizontal or vertical direction is limited to ≤90°.

[0043] It should be noted that the principles for dividing the audience's field of view can include: The horizontal and vertical directions are divided into several sectors with the front as the center. The sum of the angles of the M horizontal sectors is ≤90°, and the sum of the angles of the N vertical sectors is also ≤90°. The number of sectors divided in the two directions, the angle of each sector and even the sum of the angles may not be equal.

[0044] The partitioning of the audience's field of vision is protected by a method of dense in the middle and sparse at the edges. Since the sound and image positioning accuracy of the human auditory system in the horizontal direction in front is the highest, and the accuracy on both sides decays rapidly, the partitioning is dense in the middle and sparse at the edges, which ensures the audience experience and saves computing power. By setting the angle of the fan-shaped area in the center to no more than 5 degrees, the average virtual sound and image perception accuracy at the horizontal angle directly in front of the audience is about 4° after testing, and the more it moves to the sides, the more blurred it is. The size of the angle is set while integrating the algorithm complexity and the audience's perception accuracy to balance the two to the best.

[0045] In an optional embodiment of the present invention, obtaining the first target orientation according to the first sound and image area specifically includes: When the first sound source image is within the audience's field of view, the intersection of the angle bisector of the first mapping area on the screen is obtained, and the intersection is determined as the first target orientation.

[0046] In one embodiment, the target position calculation method may specifically include: The image recognition algorithm first gives an "actual" coordinate point (such as the center of the mouth and the center of the image as described below); then the horizontal and vertical sectors are mapped on the screen, which produces rectangular grids. The rectangular grid where the "actual" coordinate point is located is determined to be the target sound source area; then the final sound source target position within this rectangular grid is determined. The last step is to confirm the target position of the final sound source through the angular bisectors of the two sectors; It should be noted that two sectors (one horizontal and one vertical) are deduced from a certain rectangular grid, and then back to a point in the rectangular grid. Among them, the image recognition algorithm can be a mature and existing artificial intelligence image recognition technology based on algorithm models such as convolutional neural network CNN / recurrent neural network RNN, etc. The embodiment of the present invention uses the OpenCV library in python to identify and calculate the coordinate points of a certain icon in the picture.

[0047] The intersection of the angular bisectors of the mapped area on the screen is protected as the target orientation information, which is used to position the virtual sound and image, and adjust the sound position of the sound source image in real time to achieve the same position of sound and image.

[0048] In an optional embodiment of the present invention, obtaining the first target orientation according to the first sound and image area specifically includes: When the first sound source image is outside the audience's field of view, the intersection of the boundary line of the outermost sector of the audience's field of view and the screen is obtained, and the intersection is determined as the first target orientation.

[0049] In one embodiment, if Figure 4 As shown in the example of M=5, N=1, when the audience faces the edge of the screen, causing the coordinates of the sound source image on the screen to exceed the mapping area of ​​all the sectors divided by it on the screen, the midpoint of the edgemost sector boundary surface on the screen in this direction is set as the target direction for virtual sound image generation.

[0050] That is, when the first sound source image is outside the audience's field of view, the position of the virtual sound image 1 is the first target orientation.

[0051] In an optional embodiment of the present invention, obtaining the first target orientation according to the first sound and image area specifically includes: When the first sound source image is in the outermost sector of the audience's field of view, and the angle bisectors of the first mapping area have no intersection on the screen, the midpoint of the boundary of the outermost sector on the screen is determined as the first target orientation.

[0052] In one embodiment, if Figure 4As shown in the example of M=5, N=1, when the coordinates of the sound source image on the screen are located in the intersection mapping area of ​​one or two sectors, but the angular bisector of the sector does not intersect with the screen, the virtual sound image orientation corresponding to the sound source image is set to the midpoint of the edgemost sector boundary on the screen.

[0053] That is, when the first sound source image is in the outermost sector of the audience's visual field, and the angle bisector of the first mapping area has no intersection on the screen, the position of the virtual sound image 2 is the first target orientation.

[0054] In an optional embodiment of the present invention, Figure 5 As shown, after generating a virtual sound image of the first sound source image at the first target orientation and playing sound through the virtual sound image, the method further includes: S501, obtaining an angle-area mapping relationship according to the facial orientation information of the audience; the angle-area mapping relationship is a mapping relationship between the screen and the audience's field of view; S502, obtaining first position information of a first sound source image on the screen; S503, obtaining a screen area according to the first position information; S504, obtaining an angle region corresponding to the screen region according to the angle region mapping relationship and the screen region, and determining the angle region as a first sound and image region; S505, obtaining a first target orientation according to the first sound and image area; S506, generating a virtual sound image of the first sound source image at the first target direction, and playing sound through the virtual sound image; S507, detecting the position information of the first sound source image in real time; S508: When the first sound source image changes from the first position information to the second position information belonging to the first sound image area, play the sound through the virtual sound image at the first target direction.

[0055] It should be noted that, in an optional embodiment, for example, image A is at point a on the screen, point a is located in the i-th rectangular plane area of ​​the screen, and the i-th sector area (which may include horizontal and vertical sector areas) has a mapping relationship with the i-th rectangular plane area and the center field of view of the audience. That is, point a represents the first position information of image A, the i-th rectangular plane area is the screen area, and the i-th sector area is the first sound and image area.

[0056] The first sound source image changes from the first position information to the second position information belonging to the first sound image area, indicating that the sound source image moves from point a to point a1, and both point a and point a1 belong to the i-th rectangular plane area corresponding to the i-th sector area.

[0057] In an optional embodiment of the present invention, Figure 6 As shown, the generating of a virtual sound image of the first sound source image at the first target orientation and the playing of sound through the virtual sound image further includes: S601, obtaining an angle-area mapping relationship according to the facial orientation information of the audience; the angle-area mapping relationship is a mapping relationship between the screen and the audience's field of view; S602, obtaining first position information of a first sound source image on the screen; S603, obtaining a screen area according to the first position information; S604, obtaining an angle region corresponding to the screen region according to the angle region mapping relationship and the screen region, and determining the angle region as a first sound and image region; S605, obtaining a first target orientation according to the first sound and image area; S606, generating a virtual sound image of the first sound source image at the first target direction, and playing sound through the virtual sound image; S607, detecting the position information of the first sound source image in real time; S608, when the first sound source image changes from the first position information to the third position information belonging to the second sound and image area, the second target orientation is obtained according to the second sound and image area; it should be noted that, in an optional embodiment, the third position information indicates that the sound source image moves from point a to point b, point a and point b are not in the same screen area, point b is in the i-1th rectangular plane area corresponding to the i-1th sector area, and the sector area corresponding to point b has changed at this time, so the corresponding orientation information should also be adjusted with the sector area. The change in the position of the sound source image can be a change in the position of the same object, or a change in both the object and the position, for example, the sound image A moves from point a to point b, which can be a change from sound image A to sound image B, the object changes, and the corresponding position also moves from point a to point C of image B.

[0058] S609: Move the virtual sound image from the first target position to the second target position, and play sound through the virtual sound image at the second target position.

[0059] The refresh target position information is protected and used as the initial position information when the sound source image position changes next time; when the sound source image position changes again, it can be discovered in time through real-time detection, and a virtual sound image can be generated in time at the new target position to ensure that the sound source image after movement can still maintain the same position as the image.

[0060] In one embodiment, if Figure 7 As shown, a method for audio and video co-location includes: Provide the audience's initial facial orientation and vertical distance from the screen; The audience's front view is divided into several angular areas from the center to the edge in an uneven manner from small to large. It should be noted that the human auditory system has the highest accuracy in the horizontal direction of the sound and image in front, and the positioning accuracy decreases rapidly from the middle to the sides. Relevant literature shows that the average accuracy of the subjects' perception of virtual sound and image at the horizontal angle in front is about 4°, and the positioning becomes more blurred as it moves to the sides. The human auditory system is not sensitive to the movement of sound and image at height, and sometimes the sound and image movement will only be perceived when it produces an angular displacement of 60° at the subject.

[0061] In a certain embodiment, the facial orientation of the audience is monitored in real time, and the central area of ​​the audience's front field of vision is divided into angles in a non-uniform manner from the center to both sides and from small to large; M horizontal and N vertical fan-shaped areas are formed, which are dense in the middle and sparse at the edges, and M and N are limited to 10≥M≥3≥N≥1.

[0062] An angle-area mapping relationship is established on the screen; it should be noted that, in a certain embodiment, a coordinate system is established for the display area of ​​the screen, and the above-mentioned sector angle is mapped to the coordinate system of the screen according to the vertical distance from the viewer to the screen, and the coordinate range of the mapping area corresponding to each sector is confirmed, and the coordinates of the intersection of each two horizontal and vertical sector angle bisectors in the mapping area are confirmed; When the screen displays a video image, the video signal processing module gives the coordinate point of the sound source.

[0063] For the image of the speaker, give the coordinates of the center point of the mouth; For non-human sound sources, give the coordinates of the midpoint of the horizontal dimension of its image.

[0064] The video signal processing module mainly refers to the image processing software, which is used to identify the image content and give the corresponding coordinate points. For example, the currently popular artificial intelligence image recognition technology based on the algorithm model such as convolutional neural network CNN / recurrent neural network RNN ​​is an existing mature technology for this patent proposal. In practice, this patent uses the OpenCV library in Python to identify and calculate the coordinate points of an icon in the picture.

[0065] Refresh the audience's facial orientation; Determine whether the viewer's facial orientation has changed. If so, return to confirm the area in front of the viewer. If no change has occurred, the initial coordinates of the sound source image on the screen are provided; Determine in which angle mapping area the sound source image is located; The intersection of the current angle bisector on the screen is used as the target orientation to generate virtual sound images; the method for generating virtual sound images may include: calculating the different volumes of each speaker according to the angle between each speaker, the virtual sound image target orientation and the target listening point, and converting it into gain to process the audio signal to be played. Based on the existing public virtual sound image positioning algorithms such as VBAP and DBAP.

[0066] Refresh the coordinates of the sound source image on the screen; Determine whether the sound source image leaves the previous angle mapping area. If not, generate a virtual sound image at the current target position. In one embodiment, Figure 8 As shown, an example of virtual sound image generation when M=5, N=1.

[0067] If you leave the current area, the intersection of the new angle bisector on the screen is used as the target orientation to generate a virtual sound image, and the coordinates of the sound source image on the screen are refreshed.

[0068] In another embodiment of the present invention, Fig. 9 As shown, a playback device 900 for co-location of sound and image includes: a mapping relationship building module 901, a position information acquisition module 902, a mapping area acquisition module 903, a target position information acquisition module 904 and a virtual sound and image generation module 905; The mapping relationship building module 901 is used to obtain an angle-area mapping relationship according to the facial orientation information of the audience; the angle-area mapping relationship is a mapping relationship between the screen and the audience's visual field area; the position information acquisition module 902 is used to acquire the first position information of the first sound source image on the screen; The mapping area acquisition module 903 is used to obtain the screen area according to the first position information; then obtain the angle area corresponding to the screen area according to the angle area mapping relationship and the screen area, and determine the angle area as the first sound and image area; The target position information acquisition module 904 is used to obtain the first target position according to the first sound and image area; The virtual sound image generation module 905 is used to generate a virtual sound image of the first sound source image at the first target direction, and play sound through the virtual sound image.

[0069] The above embodiment provides a device for playing audio and video in the same position, which can realize the technical solution described in the above embodiment of a method for playing audio and video in the same position. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of a method for playing audio and video in the same position, which will not be repeated here.

[0070] like Fig.10As shown, the present invention also provides an electronic device 1000. The electronic device 1000 includes a processor 1001, a memory 1002 and a display 1003. Fig.10 Only some components of the electronic device 1000 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0071] In some embodiments, the memory 1002 may be an internal storage unit of the electronic device 1000, such as a hard disk or memory of the electronic device 1000. In other embodiments, the memory 1002 may also be an external storage device of the electronic device 1000, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device 1000.

[0072] Furthermore, the memory 1002 may include both an internal storage unit of the electronic device 1000 and an external storage device. The memory 1002 is used to store application software installed in the electronic device 1000 and various data.

[0073] In some embodiments, the processor 1001 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 1002, such as a method for playing audio and video in the same position in the present invention.

[0074] In some embodiments, the display 1003 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 1003 is used to display information on the electronic device 1000 and to display a visual user interface. The components 1001-1003 of the electronic device 1000 communicate with each other via a system bus.

[0075] In some embodiments of the present invention, when the processor 1001 executes the audio and video co-location playback program in the memory 1002, the following steps may be implemented: According to the facial orientation information of the audience, an angle-area mapping relationship is obtained; the angle-area mapping relationship is a mapping relationship between the screen and the audience's field of view; Acquire first position information of a first sound source image on the screen; Obtaining the screen area according to the first position information; According to the angle-area mapping relationship and the screen area, an angle area corresponding to the screen area is obtained, and the angle area is determined as a first sound-image area; Obtaining a first target orientation according to the first sound and image area; A virtual sound image of the first sound source image is generated at the first target orientation, and sound is played through the virtual sound image.

[0076] It should be understood that: when the processor 1001 executes the audio and video co-location playback program in the memory 1002, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0077] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 1000 mentioned, and the electronic device 1000 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1000 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0078] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for playing audio and video in the same position provided by the above methods is implemented, and the method comprises: According to the facial orientation information of the audience, an angle-area mapping relationship is obtained; the angle-area mapping relationship is a mapping relationship between the screen and the audience's field of view; Acquire first position information of a first sound source image on the screen; Obtaining the screen area according to the first position information; According to the angle-area mapping relationship and the screen area, an angle area corresponding to the screen area is obtained, and the angle area is determined as a first sound-image area; Obtaining a first target orientation according to the first sound and image area; A virtual sound image of the first sound source image is generated at the first target orientation, and sound is played through the virtual sound image.

[0079] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0080] The above is a detailed introduction to the audio and video co-location playback method, device and equipment provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for playing audio and video in the same position, characterized in that: include: According to the audience's facial orientation information, an angle-area mapping relationship is obtained; The angle area mapping relationship is the mapping relationship between the screen and the audience's field of view; Acquire first position information of a first sound source image on the screen; Obtaining the screen area according to the first position information; According to the angle-area mapping relationship and the screen area, an angle area corresponding to the screen area is obtained, and the angle area is determined as a first sound-image area; Obtaining a first target orientation according to the first sound and image area; A virtual sound image of the first sound source image is generated at the first target orientation, and sound is played through the virtual sound image.

2. The method for playing audio and video in the same position as claimed in claim 1, characterized in that: The step of obtaining the angle-area mapping relationship according to the facial orientation information of the audience specifically includes: Detect the audience's facial orientation in real time and obtain the audience's facial orientation information; Acquiring vertical distance data between the audience and the screen according to the facial orientation information; According to the vertical data, the audience's visual field area is symmetrically divided into a plurality of angle areas from small to large angles; An angle region mapping relationship between the screen and a plurality of angle regions is established.

3. A method for playing audio and video in the same position as claimed in claim 2, characterized in that: The method of dividing the audience's field of view into a plurality of angle areas symmetrically from small to large angles according to the vertical data specifically includes: The audience's visual field is divided into M horizontal sector-shaped areas and N vertical sector-shaped areas in a centrally symmetrical manner; wherein M and N are odd numbers respectively; The angle of the fan-shaped area close to the center is no more than 5 degrees, and the angles of other fan-shaped areas increase gradually from the center to the edge; the audience field of view is a fan-shaped area with a centrally symmetrical angle of 60 degrees.

4. The method for playing audio and video in the same position as claimed in claim 1, characterized in that: The obtaining of the first target position according to the first sound and image area specifically includes: When the first sound source image is within the audience's field of view, the intersection of the angle bisector of the first mapping area on the screen is obtained, and the intersection is determined as the first target orientation.

5. The method for playing audio and video in the same position as claimed in claim 1, characterized in that: The obtaining of the first target position according to the first sound and image area specifically includes: When the first sound source image is outside the audience's field of view, the intersection of the boundary line of the outermost sector of the audience's field of view and the screen is obtained, and the intersection is determined as the first target orientation.

6. The method for playing audio and video in the same position as claimed in claim 1, characterized in that: The obtaining of the first target position according to the first sound and image area specifically includes: When the first sound source image is in the outermost sector of the audience's field of view, and the angle bisectors of the first mapping area have no intersection on the screen, the midpoint of the boundary of the outermost sector on the screen is determined as the first target orientation.

7. The method for playing audio and video in the same position as claimed in claim 1, characterized in that: After generating a virtual sound image of the first sound source image at the first target position and playing sound through the virtual sound image, the method further includes: Detecting the position information of the first sound source image in real time; When the first sound source image changes from the first position information to the second position information belonging to the first sound image area, sound is played through the virtual sound image at the first target position.

8. The method for playing audio and video in the same position as claimed in claim 1, characterized in that: After generating a virtual sound image of the first sound source image at the first target position and playing sound through the virtual sound image, the method further includes: Detecting the position information of the first sound source image in real time; When the first sound source image changes from the first position information to the third position information belonging to the second sound image area, obtaining a second target orientation according to the second sound image area; The virtual sound image is moved from the first target position to the second target position, and sound is played through the virtual sound image at the second target position.

9. A playback device for audio and video co-location, characterized in that: include: A mapping relationship building module, a position information acquisition module, a mapping area acquisition module, a target position information acquisition module and a virtual sound and image generation module; The mapping relationship building module is used to obtain an angle-area mapping relationship according to the face orientation information of the audience; the angle-area mapping relationship is a mapping relationship between the screen and the audience's field of view; The position information acquisition module is used to acquire first position information of a first sound source image on the screen; The mapping area acquisition module is used to obtain the screen area according to the first position information; Then, according to the angle-area mapping relationship and the screen area, an angle area corresponding to the screen area is obtained, and the angle area is determined as a first sound-image area; The target position information acquisition module is used to obtain the first target position according to the first sound and image area; The virtual sound image generation module is used to generate a virtual sound image of the first sound source image at the first target orientation, and to play sound through the virtual sound image.

10. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the audio and video co-location playback method described in any one of claims 1 to 7.

Images