Sound positioning and displaying method and system based on 360-degree panoramic video
Through the fusion of ring microphone array and multi-visual features, high-precision sound source positioning and display of 360-degree panoramic video is achieved, which solves the problem of insufficient sound source positioning accuracy and single audio-visual fusion strategy in traditional methods, and improves user experience and device utilization rate .
Patent Information
- Application Number
- CN202411015583.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-22
AI Technical Summary
In terms of sound acquisition and display, traditional 360-degree panoramic videos have user experience problems caused by insufficient sound source positioning accuracy, single audio-visual fusion strategy, and device restrictions, making it difficult to take into account the display of panoramic vision and local details.
A ring microphone array is used to collect multiple audio signals, divide fine-grained sectors through interpolation expansion, combine the active areas, texture features and facial visual clues of multi-frame video images to perform weighted fusion positioning, and adopt a panoramic + close-up display method , seamlessly stitch the 360-degree panorama with close-up images.
It significantly improves the accuracy and angular resolution of sound source positioning, realizes synchronization between sound source and video target, improves the user's immersion and substitution, adapts to complex scenarios, flexible fusion strategies and efficient algorithms are suitable for embedded systems .
Smart Images

Figure CN119011950A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of sound positioning and display methods, and more specifically, to a sound positioning and display method and system based on 360-degree panoramic video. Background Art
[0002] With the development of multimedia technology, 360-degree panoramic video has been widely used in virtual reality, remote monitoring, live broadcast and other fields due to its wide field of view and immersive experience. Traditional 360-degree panoramic video stitching methods usually use multiple cameras in a circular array to synchronously capture images, and then stitch these images into a 360-degree panorama through image matching and fusion algorithms.
[0003] In terms of sound collection, traditional methods mostly use a single microphone to record panoramic sound, or use several microphones to record sounds from different directions and then mix them together. This approach ignores the spatial distribution characteristics of sound and cannot accurately locate the direction of the sound source, resulting in asynchrony between sound and picture, affecting user experience.
[0004] To solve this problem, some researchers have proposed a sound source localization method based on microphone arrays. This method uses multiple microphones to record sound signals and estimates the spatial position of the sound source by analyzing the time delay and amplitude differences between the microphone signals. However, microphone array sound source localization is limited by the number and distribution of microphones, and the positioning accuracy and resolution are difficult to meet the requirements. In addition, the simple sound source localization result lacks correlation with the 360-degree panoramic video and cannot intuitively reflect the position of the sound source in the video.
[0005] Some methods try to combine sound source localization with video analysis, by detecting visual clues such as moving objects and faces in the video to assist in determining the location of the sound source. However, most of these methods are based on a single visual feature, lack multi-feature fusion, and are not adaptable to complex scenes. Moreover, most existing audio-visual fusion methods use a simple weighted average, with a single fusion strategy and a lack of flexibility in weight allocation, making it difficult to cope with dynamically changing scenes.
[0006] In terms of panoramic video display, traditional methods are limited by the resolution and aspect ratio of the display device, and it is often difficult to take into account both the 360-degree panorama and local details. Some methods use fisheye lenses to display 360-degree panoramas, but the image is severely distorted, details are lost, and the user experience is poor. Other methods use multiple tiled ordinary lenses to stitch together, but the stitching gaps are obvious and the coherence is poor. How to display a 360-degree panorama and highlight the details of the area of interest within a limited screen size is an urgent problem to be solved.
[0007] In view of the above problems, the present invention proposes a sound localization and display method based on 360-degree panoramic video. The method first collects multi-channel audio signals through a circular microphone array, and divides the 360-degree space into more fine-grained sectors using the interpolation and expansion method, which significantly improves the angular resolution of sound source localization. On this basis, multiple visual clues such as activity areas, texture features, and faces of multi-frame video images are extracted, and they are combined with sound information through a weighted fusion strategy to obtain a more robust and reliable sound source localization result.
[0008] In terms of video display, the present invention adopts a "panorama + close-up" approach, compressing the 360-degree panoramic image into the upper area, and dynamically displaying the close-up image extracted according to the sound source positioning result in the lower area. The two are seamlessly spliced, which not only ensures the panoramic field of view, but also highlights the local details. At the same time, the audio-visual information is closely linked, so that the sound source and the video target are synchronized, greatly enhancing the user's immersion and sense of substitution. Summary of the invention
[0009] In view of this, the present invention proposes a sound localization and display method based on 360-degree panoramic video to solve the above problems. The method first collects multi-channel audio signals through a circular microphone array, and divides the 360-degree space into more fine-grained sectors using the interpolation and expansion method, which significantly improves the angular resolution of sound source localization. On this basis, multiple visual clues such as the activity area, texture features, and faces of multiple frames of video images are extracted, and they are combined with sound information through a weighted fusion strategy to obtain a more robust and reliable sound source localization result. In terms of video display, the present invention adopts the "panoramic + close-up" method to compress the 360-degree panoramic image into the upper area, and the lower area dynamically displays the close-up image extracted according to the sound source localization result. The two are seamlessly spliced, which not only ensures the panoramic field of view, but also highlights the local details. At the same time, the audio-visual information is closely associated, so that the sound source and the video target are synchronized, which greatly enhances the user's immersion and sense of substitution.
[0010] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0011] A sound positioning and display method based on 360-degree panoramic video, comprising:
[0012] The step of synchronously calibrating the microphone array and the panoramic video lens;
[0013] Steps for cropping, scaling and fusing panoramic videos;
[0014] Steps for collecting audio data and expanding microphone azimuth;
[0015] The steps of locating the sound source and generating a sound histogram;
[0016] The step of detecting active pixels and texture regions in the video and generating a visual activity histogram;
[0017] The step of fusing the sound histogram and the visual activity histogram and establishing a mapping of azimuth angles to pixel coordinates;
[0018] Steps for extracting close-up images according to the azimuth angle of the sound source;
[0019] Steps for stitching and displaying a panoramic image and a close-up image.
[0020] Specifically, the step of synchronously calibrating the microphone array and the panoramic video lens includes:
[0021] Take one microphone physical channel in the microphone array as the 0 degree reference, and mark the remaining microphone physical channels in sequence in the same direction;
[0022] Take one shot in the panoramic video as the 0-degree reference and mark the remaining shots in sequence in the same direction as the microphone array.
[0023] Specifically, the steps of cropping, scaling and fusing the panoramic video include:
[0024] Stitch four 1920*1080 resolution panoramic videos into a 7680*1080 image;
[0025] Crop the overlapping area of the stitched images to obtain a panoramic image with a resolution of 6000*840;
[0026] The 6000*840 panoramic image is scaled to 1920*1080, of which the 1920*270 area is used as the effective panoramic area and the 1920*810 area is used to place the close-up image.
[0027] Specifically, the step of collecting audio data and expanding the microphone azimuth includes: collecting audio data through a circular array composed of 8 microphones, the microphones are evenly distributed on the circumference, and the angle between adjacent microphones is 45°; dividing 360° into 32 sectors, each sector corresponding to an azimuth angle θ i (i=1,2,...,32), the angle between adjacent azimuths is about 11.25°.
[0028] Specifically, the steps of locating the sound source and generating a sound histogram include: performing energy calculation on the audio data collected at 32 azimuth angles to obtain corresponding energy values E i (i=1,2,...,32); according to the energy value E i Generate a sound source histogram H1(θ)=(θ1,E1),(θ2,E2),...,(θ 32 ,E32 ), revealing the distribution of sound in all directions.
[0029] Specifically, the step of detecting active pixels and texture areas in the video and generating a visual activity histogram includes calculating the difference between the current frame image I(x,y) and the background frame image B(x,y) δ(x,y)=I(x,y)-B(x,y)|, if δ(x,y) is greater than a threshold T, it is determined to be an active pixel; judging whether the pixel belongs to a specific texture interval according to the YCbCr value of the pixel, forming a texture mask map M(x,y); weighted fusion of the active pixel map and the texture mask map to obtain a visual activity map V(x,y)=w m M(x,y)+w a A(x,y); calculate the cumulative value V of the visual activity in the sector corresponding to each azimuth angle θ i , generate visual activity histogram H2(θ)=(θ1,V1),(θ2,V2),...,(θ 32 ,V 32 ).
[0030] Specifically, the step of fusing the sound histogram and the visual activity histogram and establishing a mapping from azimuth to pixel coordinates includes synthesizing the sound histogram H1(θ) and the visual activity histogram H2(θ) by a weighted fusion method to obtain a joint sound source azimuth histogram H(θ): H(θ)=w1·H1(θ)+w2·H2(θ)
[0031] Wherein, w1 and w2 are weight coefficients of sound and visual activity, respectively, and w1+w2=1;
[0032] Establish a mapping relationship between the 360° azimuth angle θ and the panoramic video width pixel W
[0033] x=f(θ)=[(θ+180°)] / 360°·W
[0034] Where x is the horizontal coordinate of the pixel corresponding to the azimuth angle θ, and || represents the rounding operation.
[0035] Specifically, the step of extracting a close-up image according to the azimuth angle of the sound source includes detecting the global peak of the joint sound source azimuth histogram H(θ) to obtain the azimuth angle θ of the sound source. s ; Calculate θ based on the mapping relationship from azimuth to pixel coordinates s The corresponding horizontal coordinate x s =f(θ s ); x s As the center, in its neighborhood [x s -Δx,x s +Δx] and enlarged to a resolution of 1920×810.
[0036] Specifically, the step of stitching and displaying the panoramic image and the close-up image includes stitching the panoramic image with a resolution of 1920×270 and the close-up image with a resolution of 1920×810 in a vertical direction to form a final display image of 1920×1080 and outputting the stitched display image to a 16:9 display screen.
[0037] A sound positioning and display system based on 360° panoramic video, comprising: four panoramic video acquisition modules with a resolution of 1920×1080, used to acquire 360° video; a circular sound acquisition array composed of 8 microphones, used to acquire multi-directional audio; a central processing unit, used for the sound positioning and display method; an image stitching and output module, used for stitching and displaying panoramic images and close-up images; the panoramic video acquisition module, the sound acquisition array, the central processing unit, the image stitching and output module are connected via a data bus to form a complete sound positioning and display system.
[0038] Compared with the prior art, the beneficial effects of the present invention are:
[0039] 1. The use of a circular microphone array and azimuth subdivision strategy significantly improves the accuracy and angular resolution of sound source localization, achieving detailed characterization of the sound source orientation.
[0040] 2. Fully explore the visual clues such as activity areas, texture features, faces, etc. in multi-frame videos, and combine them with sound information for multi-feature fusion positioning, thereby improving the robustness and reliability of sound source positioning in complex scenes.
[0041] 3. The fusion strategy is flexible and can dynamically adjust the weights of various audio-visual features according to the characteristics of the scene to adapt to changing application environments.
[0042] 4. The "panorama + close-up" splicing display mode realizes seamless switching between panoramic preview and detail observation under limited screen size, and the close-up image can keep up with the dynamic changes of the sound source position, highlighting the relevance of audio-visual information and bringing an immersive experience.
[0043] 5. It can make full use of existing hardware equipment, such as ordinary cameras, microphones, display screens, etc., without the need for expensive special equipment, and is easy to promote and apply.
[0044] 6. The software algorithm is efficient and practical, suitable for real-time operation in embedded systems, and has strong customizability and scalability, which provides convenience for subsequent improvements and upgrades.
[0045] In summary, the present invention comprehensively optimizes and innovates 360-degree panoramic video from multiple levels, such as multi-channel audio pickup, multi-visual feature analysis, sound and video fusion, and panoramic close-up stitching, forming a complete, low-cost, and easy-to-deploy 360-degree panoramic sound source positioning and display solution, which is expected to be widely used in virtual reality, remote monitoring, video conferencing, live broadcasting, and other fields, and has good application prospects and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A schematic diagram of an eight-way microphone ring array of the present invention;
[0047] Figure 2 A schematic diagram of microphones at 32 azimuth angles after interpolation of the present invention;
[0048] Figure 3 It is a statistical conceptual diagram of the sound source histogram of the present invention;
[0049] Figure 4 It is a calibration flow chart of a sound positioning and display method based on 360-degree panoramic video of the present invention;
[0050] Figure 5 The present invention is a flow chart of a method for sound localization and display based on 360-degree panoramic video. DETAILED DESCRIPTION
[0051] The scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. The embodiments in the present invention and all other embodiments obtained by ordinary persons in the art without making creative work are within the scope of protection of the present invention.
[0052] In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", and "top / bottom" indicate the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.
[0053] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "set / connected", and "connected" should be understood in a broad sense. For example, "connected" means fixed connection, detachable connection, or integral connection; mechanical connection, electrical connection; direct connection, indirect connection through an intermediate medium, and internal communication between two components. For ordinary people in the field, the specific meanings of the above terms in the present invention are understood according to specific circumstances.
[0054] The present invention relates to a method and system for sound positioning and display based on 360-degree panoramic video. The 360-degree panoramic video in the prior art is usually spliced by multiple lenses, and its video aspect ratio does not match the conventional display, resulting in a poor viewing experience. At the same time, the existing panoramic video lacks the sound positioning function and cannot highlight the detailed image of the area where the sound source is located. The present invention aims to solve the above problems and provide a 360-degree panoramic video processing solution that can accurately locate the sound source position and automatically extract close-up images.
[0055] like Figure 1-5 As shown, the sound positioning and display system of the present invention includes four panoramic video acquisition modules, a circular sound acquisition array composed of 8 microphones, a central processing unit, and an image stitching and output module. Among them, the panoramic video acquisition module adopts a resolution of 1920*1080 and is used to collect images within a 360-degree viewing angle; the 8 microphones in the circular sound acquisition array are evenly distributed on a circle with a radius of r, and the angle between adjacent microphones is 45 degrees, which is used to collect audio signals in different directions; the central processing unit is used to execute the sound positioning and video processing algorithm; the image stitching and output module is used to stitch the panoramic image and the close-up image together and display them.
[0056] The workflow of this embodiment is as follows:
[0057] Step 1: Microphone array and panoramic video lens calibration
[0058] Due to the different geometric layouts of the microphone array and the panoramic video lens, there is a deviation in the sound and image orientations collected by the two. To ensure the accuracy of the positioning results, it is necessary to calibrate them in advance. The specific method is to select one element in the microphone array and the panoramic lens array as the 0° reference, and mark the orientations of the remaining elements respectively. Among them, the orientation of the microphone is marked as Mic i (i=0,1,...,7), the position of the lens is marked as Cam j (j=0,1,2,3).
[0059] Step 2: Capture and preprocess panoramic video
[0060] Four 1920×1080 resolution lenses synchronously capture images to obtain a 360° panoramic image with a total resolution of 7680×1080. Since there is a certain overlap between adjacent lenses, image fusion and cropping are required. First, the stitching gaps of the four sub-images are removed, and then the overlapping parts of the edges are cropped to obtain a panoramic image with a resolution of 6000×840. Then, the image is scaled to a resolution of 1920×1080 as input for subsequent processing. Among them, the 1920×270 area retains the complete 360° panoramic information, and the 1920×810 area is used to place close-up images.
[0061] Step 3: Sound collection and azimuth expansion
[0062] The microphone array collects 8 audio signals with a sampling rate of 48kHz and a sampling accuracy of 16bit. In order to improve the angular resolution of azimuth positioning, the 360° azimuth is divided into 32 sectors, each sector corresponding to an angle of 11.25°. That is:
[0063] θ i =(i-1)*11.25°,i=1,2,...,32
[0064] where θ i Indicates the starting angle of the ith sector. The audio signals collected by the 8 microphones are extended to 32 azimuths by interpolation to obtain 32 virtual audio signals.
[0065] Step 4: Sound source localization and histogram generation
[0066] The short-time energy of 32 audio signals is calculated to obtain the energy envelope curve E. i (n), where n is the time frame index. Take the 32 energy values at each moment and generate a histogram of the sound source direction:
[0067] H1(θ i )=E i (n),i=1,2,...,32
[0068] The histogram reflects the intensity distribution of sound in various directions, and the peak corresponds to the azimuth where the sound source is most likely to appear.
[0069] Step 5: Video active area detection and histogram generation Active area detection is performed on the video frame image I(x,y), where x and y are pixel coordinates. The detection process includes two parts: (1) Detect motion area using background difference method: Subtract the current frame from the background frame to obtain a difference map:
[0070] D(x,y)=|I(x,y)-b(x,y)|
[0071] Where B(x,y) is the background image. Figure 2 Value it and get the motion area mask:
[0072]
[0073] Where T m is the motion detection threshold.
[0074] (2) Skin color detection:
[0075] Convert the image to YCbCr color space and extract the skin color area according to the distribution range of skin color
[0076]
[0077] where Y min′ Y max′ Cb min′ Cb max′ Cr min′ Cr max The upper and lower thresholds of the skin color component. Perform an “or” operation on the motion mask image and the skin color mask image to obtain the final active area mask image:
[0078] A(x,y)=M(x,y)|S(x,y)
[0079] Perform azimuth projection on the active area mask image A(x,y) to obtain the video active area azimuth histogram:
[0080] H2(θ i )=∑A(x,y),i=1,2,...,32
[0081] where (x,y)∈θ i Indicates that the pixel (x, y) falls within the i-th azimuth sector.
[0082] Step 6: Sound-visual information fusion and sound source localization
[0083] The sound histogram H1(θ) and the visual activity area histogram H2(θ) are combined using the weighted fusion method to obtain the final sound source orientation probability distribution:
[0084] H(θ)=w1*H1(θ)+w2*H2(θ)
[0085] Where w1 and w2 are weight coefficients, satisfying w1+w2=1. Find the azimuth angle θ corresponding to the maximum probability value in H(θ) s , which is the estimated direction of the sound source.
[0086] Step 7: Close-up image extraction and scaling according to the sound source azimuth angle θ sCalculate the corresponding pixel coordinates in the panorama:
[0087] x s =(θ s / 360°)*W
[0088] Where W is the width of the panorama. s As the center, expand Δx pixels to the left and right to obtain the boundary coordinates of the sound source area:
[0089] x1=x s -Δx,x2=x s +Δx
[0090] When x1<0, x1=0; when x2>W, x2=W. Extract the sound source area image from the [y1,y2] row and [x1,x2] column of the panorama and scale it to a resolution of 1920×810, where:
[0091] y1=(H-810) / 2,y2=y1+810
[0092] H is the panorama height.
[0093] Step 8: Stitching and displaying the panoramic image and close-up image
[0094] The 270-line panoramic image and the 810-line close-up image are vertically spliced to obtain the final 1920*1080 resolution display image, which is then output to the display screen.
[0095] This embodiment comprehensively utilizes multiple features of sound signals and video images, collects multi-directional audio through a circular microphone array, and estimates the sound source direction using a sound intensity azimuth histogram; extracts active areas in the video through motion detection and skin color detection, and uses a visual activity azimuth histogram to supplement and calibrate the sound localization result; performs weighted fusion of the two histograms to obtain a more robust and reliable sound source localization result; extracts the sound source area image from the 360-degree panoramic video based on the localization result, and splices it with the panoramic background image for display, ultimately obtaining a picture presentation effect with both a global overview and local details, providing users with an immersive experience.
[0096] It should be noted that the parameter values in this embodiment are only exemplary settings and do not constitute a limitation on the range of parameter values. For example, the number of microphones is not limited to 8, but can also be 16, 32, etc.; the number of sectors is not limited to 32, and can be flexibly set according to the requirements of positioning accuracy; the image size can be adjusted according to the size and resolution of the display screen; the video activity area detection algorithm is not limited to background difference and skin color detection, and other motion detection, saliency detection, semantic segmentation and other algorithms can be selected; the weight of sound-visual information fusion can be dynamically adjusted according to the application scenario and environmental conditions to achieve the best positioning effect.
[0097] Example 2
[0098] In some application scenarios, such as concerts and lecture halls, the sound source area is usually dominated by human voices. At this time, face detection technology can be used to further refine the sound source localization results.
[0099] In some application scenarios, such as concerts and lecture halls, the sound source area is usually dominated by human voices. In this case, face detection technology can be used to further refine the sound source localization results. In step 5, the Adaboost cascade classifier based on Haar features is used to detect faces in the video frame images, and a set of face rectangle frames {(x k ,y k ,w k ,h k )}, where (x k ,y k ) is the coordinate of the upper left corner of the kth face frame, w k and h k are the width and height of the face frame, respectively, k=1,2,...,K, where K is the number of faces detected. Calculate the center coordinates of each face frame:
[0100]
[0101] According to the direction of the line from the center coordinate of the face frame to the center of the panorama, it is quantized to 32 azimuth sectors to obtain the face azimuth histogram:
[0102]
[0103] Where #{} represents the number of elements in the collection.
[0104] In step 6, the face orientation histogram H3(θ) is combined with the sound histogram H1(θ) and the visual activity area histogram H2(θ) to obtain the final sound source orientation probability distribution:
[0105] H(θ)=w1*H1(θ)+w2*H2(θ)+w3*H3(θ)
[0106] Where w3 is the weight coefficient of the face azimuth histogram, w1+w2+w3=1. Considering that the sound source corresponding to the face area is more likely, w3 can be given a higher weight. In terms of close-up image extraction, the detected face frame is expanded outward by a certain proportion to obtain the sound source close-up area. Assuming that the scaling ratio of the close-up image to the original panoramic image is s, the width and height of the face frame in the close-up image are:
[0107]
[0108] The boundary coordinates of the close-up area are:
[0109]
[0110] Where α is the expansion factor, which is generally 0.2-0.5, indicating that the area is expanded by 20%-50% around the face frame as a close-up image. In the case of multiple face frames, their circumscribed rectangles are taken as close-up areas. If the close-up area exceeds the panoramic image range, it is truncated to the border of the panoramic image. This embodiment uses face detection technology to extract the area where the sound source may appear, introduces the face orientation information into the decision-making process of sound source positioning, and expands the close-up area based on the face frame, which not only highlights the semantically key area of the face, but also covers other sound sources that may exist near the face, making the close-up image richer in semantic information and in line with people's audio-visual habits.
[0111] Example 3
[0112] In other application scenarios, such as sports stadiums and stage plays, the sound source area may be accompanied by large-scale movements. At this time, motion target detection and tracking technology can be used to achieve dynamic positioning and close-up capture of the sound source.
[0113] In step 5, the frame difference method is used to detect moving targets in the video sequence. The difference between two adjacent frame images I(x, y, t) and I(x, y, t-1) is obtained to obtain the difference image
[0114] D(x,y,t)=|I(x,y,t)-I(x,y,t-1)|
[0115] Set the threshold T d Difference Figure 2 Valued, get the moving target mask:
[0116]
[0117] Perform a connected domain analysis on the moving target mask image to obtain the external rectangular frame of N moving targets {(x n ,y n ,w n ,hn ,t)}, where (x n ,y n ) is the coordinate of the upper left corner of the nth target box, w n and h n are the width and height of the target box respectively, t is the current moment, n=1,2,...,N, and the Kalman filter algorithm is used to track and predict its position and speed at the next moment.
[0118] The target position and velocity are updated according to the following equations:
[0119]
[0120] Where Δt is the time interval between two adjacent frames. and are the speeds of the nth target in the x and y directions respectively. Calculate its azimuth according to the center coordinates of the target frame:
[0121]
[0122] Perform histogram statistics on the azimuth angles of all moving targets to obtain the moving target azimuth angle histogram:
[0123] H4(θ,t)=#θ n (t)∈θ,θ=1°,2°,...,360°
[0124] In the sound source localization of step 6, the moving target azimuth histogram H4(θ,t) is used as an additional decision basis, together with the sound histogram, the visual activity area histogram, and the face azimuth histogram. Figure 1 Start weighted fusion:
[0125] H(θ,t)=w1*H1(θ,t)+w2*H2(θ,t)+w3*H3(θ,t)+w4*H4(θ,t)
[0126] Where w4 is the weight coefficient of the moving target azimuth histogram, w1+w2+w3+w4=1. The weight setting can be dynamically adjusted according to the motion characteristics of the sound source in the scene. For example, the weight of w4 can be appropriately increased in a scene with intense motion.
[0127] In terms of close-up image extraction, based on the moving target frame, the size and position of the close-up area are adjusted according to its moving speed and direction, so that the close-up image can cover the moving trajectory of the sound source. Assume that a sound source moving target is detected at time t, and its circumscribed rectangular frame is (x s (t),y s (t),w s (t),h s (t)), the corresponding rectangular box in the close-up image is Then there are:
[0128]
[0129] Where β is the scaling factor from motion speed to close-up size and are the speeds of the sound source target in the x and y directions respectively, and Δt is the time advance for taking the close-up image. The greater the speed and the time advance, the larger the close-up area will be to accommodate the rapid movement of the sound source. At the same time, the position of the close-up area will also be offset according to the speed direction, making the content of the close-up image more advanced and reducing the possibility of the moving target moving out of the picture.
[0130] In the case of multiple moving targets, the target corresponding to the azimuth of the strongest sound source is selected as the close-up object. To avoid frequent jumps in the close-up area, the time lag and spatial neighborhood restrictions of the target selection can be set to ensure a smooth transition of the close-up image.
[0131] This embodiment uses motion target detection and tracking technology to predict the motion trend of the sound source target in real time, and feeds back the motion information to the process of sound source positioning and close-up extraction, overcoming the lag of traditional positioning methods and the limitations of close-up image content. By adaptively adjusting the size and position of the close-up area, the close-up image always follows the motion changes of the sound source target while maintaining stability and continuity, greatly improving the dynamic performance of sound source positioning and close-up extraction, and providing users with a more intelligent and smooth audio-visual experience.
[0132] Example 3
[0133] The present invention also discloses a sound positioning and display system based on 360° panoramic video, comprising: four panoramic video acquisition modules with a resolution of 1920×1080, used to acquire 360° video; a circular sound acquisition array composed of 8 microphones, used to acquire multi-directional audio; a central processing unit, used to execute the sound positioning and display method; an image stitching and output module, used to stitch and display panoramic images and close-up images; the panoramic video acquisition module, the sound acquisition array, the central processing unit, the image stitching and output module are connected via a data bus to form a complete sound positioning and display system.
[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A sound positioning and display method based on 360-degree panoramic video, Features ,include: The step of synchronously calibrating the microphone array and the panoramic video lens; Steps for cropping, scaling and fusing panoramic videos; Steps for collecting audio data and expanding microphone azimuth; The steps of locating the sound source and generating a sound histogram; The step of detecting active pixels and texture regions in the video and generating a visual activity histogram; The step of fusing the sound histogram and the visual activity histogram and establishing a mapping of azimuth angles to pixel coordinates; Steps for extracting close-up images according to the azimuth angle of the sound source; Steps for stitching and displaying a panoramic image and a close-up image.
2. The method according to claim 1, Features , the step of synchronously calibrating the microphone array and the panoramic video lens includes: Take one microphone physical channel in the microphone array as the 0 degree reference, and mark the remaining microphone physical channels in sequence in the same direction; Take one shot in the panoramic video as the 0-degree reference and mark the remaining shots in sequence in the same direction as the microphone array.
3. The method according to claim 1, Features , the steps of cropping, scaling and fusing the panoramic video include: Stitch four 1920*1080 resolution panoramic videos into a 7680*1080 image; Crop the overlapping area of the stitched images to obtain a panoramic image with a resolution of 6000*840; The 6000*840 panoramic image is scaled to 1920*1080, of which the 1920*270 area is used as the effective panoramic area and the 1920*810 area is used to place the close-up image.
4. The method according to claim 1, It is characterized in that The step of collecting audio data and expanding the microphone azimuth includes: collecting audio data through a circular array composed of 8 microphones, the microphones are evenly distributed on the circumference, and the angle between adjacent microphones is 45°; dividing 360° into 32 sectors, each sector corresponding to an azimuth angle θ i (i=1,2,...,32), the angle between adjacent azimuth angles is about 11.25°.
5. The method according to claim 1, It is characterized in that The steps of locating the sound source and generating a sound histogram include: performing energy calculation on the audio data collected at 32 azimuth angles to obtain corresponding energy values E i (i=1,2,...,32); according to the energy value E i Generate sound source histogram H 1 (θ)=(θ 1 , E 1 ), (θ 2 , E 2 ),...,(θ 32 , E 32 ), revealing the distribution of sound in all directions.
6. The method according to claim 1, It is characterized in that The step of detecting active pixels and texture areas in the video and generating a visual activity histogram comprises calculating the difference δ(x, y)=|I(x, y)-B(x, y)| between the current frame image I(x, y) and the background frame image B(x, y), and if δ(x, y) is greater than a threshold T, the active pixel is determined to belong to a specific texture interval according to the YCbCr value of the pixel to form a texture mask map M(x, y); weighted fusion of the active pixel map and the texture mask map to obtain a visual activity map V(x, y)=w m M(x, y)+W a A(x, y); calculate the cumulative value V of the visual activity in the sector corresponding to each azimuth angle θ i , generate the visual activity histogram H 2 (θ)=(θ 1 , V 1 ), (θ 2 , V 2 ),...,(θ 32 , V 32 ).
7. The method according to claim 1, It is characterized in that The step of fusing the sound histogram and the visual activity histogram and establishing a mapping from azimuth to pixel coordinates includes synthesizing the sound histogram H by a weighted fusion method. 1 (θ) and visual activity histogram H 2 (θ), and obtain the joint sound source orientation histogram H(θ): H(θ)=W 1 ·H 1 (θ)+W 2 ·H 2 (θ) Among them, w 1 and w 2 are the weight coefficients of sound and visual activity respectively, and w 1 +W 2 =1; Establish a mapping relationship between the 360° azimuth angle θ and the panoramic video width pixel W x=f(θ)=[(θ+180°)] / 360°·W Where x is the horizontal coordinate of the pixel corresponding to the azimuth angle θ, and || represents the rounding operation.
8. The method according to claim 7, It is characterized in that The step of extracting a close-up image according to the azimuth angle of the sound source includes detecting the global peak of the joint sound source azimuth histogram H(θ) to obtain the azimuth angle θ of the sound source. s ; Calculate θ based on the mapping relationship from azimuth to pixel coordinates s The corresponding horizontal coordinate x s =f(θ s ); x s As the center, in its neighborhood [x s -Δx,x s +Δx] and enlarged to a resolution of 1920×810.
9. The method according to claim 1, It is characterized in that The step of stitching and displaying the panoramic image and the close-up image includes stitching the panoramic image with a resolution of 1920×270 and the close-up image with a resolution of 1920×810 in a vertical direction to form a final display image of 1920×1080 and outputting the stitched display image to a 16:9 display screen.
10. A sound positioning and display system based on 360° panoramic video, It is characterized in that include: Four panoramic video acquisition modules with a resolution of 1920×1080, used to acquire 360° videos; a circular sound acquisition array composed of 8 microphones, used to acquire multi-directional audio; a central processing unit, used to execute the sound positioning and display method described in any one of claims 1 to 9; an image stitching and output module, used for stitching and displaying panoramic images and close-up images; the panoramic video acquisition module, the sound acquisition array, the central processing unit, the image stitching and output module are connected via a data bus to form a complete sound positioning and display system.
Citation Information
Cited By
Sound source three-dimensional positioning system and method based on binocular acoustic camera
CN121325104A
An oil and gas well casing outside optical fiber azimuth detection device and method
CN122669968A
