Audio and video processing method and audio and video processing device

Abstract

This invention proposes an audio-visual processing method and an audio-visual processing apparatus. The method includes receiving multiple initial videos and audio signals of each initial video via the audio-visual processing apparatus; generating at least one output video signal based on the multiple initial videos via the audio-visual processing apparatus; adjusting each audio signal to a stereo audio signal based on layout information via the audio-visual processing apparatus; and outputting at least one output video signal and stereo audio signals to an audio-visual playback device via the audio-visual processing apparatus, such that the audio-visual playback device outputs a display screen having multiple initial videos on multiple display blocks corresponding to the layout information by playing at least one output video signal.

Description

Technical Field

[0001] The present invention relates to an audio and video processing method and an audio and video processing device, and in particular to an audio and video processing method with audio spatialization and an audio and video processing device. Background Art

[0002] With the development of technology, today's business and entertainment models are different from the past. The changes in business and entertainment models also directly affect people's lives. For example, thanks to the development of Internet technology and the powerful audio and video playback functions of electronic products, a wave of online live broadcasting has been formed, and a large number of online live broadcasting platforms and online live broadcasting programs have emerged. Users can watch the live broadcaster's video and listen to the live broadcaster's voice through electronic products, and according to different live broadcast content, people can enjoy many different entertainment forms.

[0003] However, when users play the video and sound of live programs through electronic products (such as televisions, computers, mobile phones or tablets, etc.), the electronic products are unable to produce spatialized stereo sound effects according to the speaking position of the live broadcaster. Users cannot have a sense of presence when watching the live broadcast, which greatly reduces the user experience. Summary of the invention

[0004] In view of the above problems, an object of the present invention is to provide an audio-visual processing device, and in particular, an audio-visual processing device that can make listeners feel more immersive.

[0005] One embodiment provides an audio-visual processing device, which is suitable for connecting to an audio-visual playback device. The audio-visual processing device includes an image processing unit, a sound processing unit, and an output port. The sound processing unit is connected to the image processing unit. The output port couples the image processing unit and the sound processing unit. The image processing unit receives a plurality of initial videos and generates at least one output video signal according to the plurality of initial videos. The sound processing unit receives layout configuration information and an audio signal corresponding to each initial video, and adjusts each audio signal to a stereo audio signal according to the layout configuration information. The output port outputs the stereo audio signal and the output video signal to the audio-visual playback device, so that the audio-visual playback device outputs a display screen having a plurality of initial videos on a plurality of display blocks corresponding to the layout configuration information by playing the output video signal.

[0006] An embodiment provides an audio spatialization audio and video processing method, which is applicable to an audio and video processing device, and the audio and video processing device is connected to an audio and video playback device. Here, the audio and video processing method includes: receiving a plurality of initial videos and audio signals corresponding to each initial video through the audio and video processing device; generating at least one output video signal according to the plurality of initial videos through the audio and video processing device; adjusting each audio signal to a stereo audio signal according to a layout configuration information through the audio and video processing device; and outputting at least one output video signal and the stereo audio signal to the audio and video playback device through the audio and video processing device, so that the audio and video playback device outputs a display screen having the plurality of initial videos on a plurality of display blocks corresponding to the layout configuration information by playing the at least one output video signal.

[0007] In summary, according to any embodiment of the audio-visual processing method and the audio-visual processing device of the present invention, the audio-visual processing device can adjust the signal parameters of the audio signal of the video according to the display position of the video. In some embodiments, the stereo audio signal has a difference in volume, delay, phase, frequency or other signal parameters in any combination compared to the audio signal; or the stereo audio signal corresponds to each channel of the audio-visual playback device, and when the stereo audio signal is output on different channels, it is different from each other in volume, delay, phase, frequency or other signal parameters in any combination. In this way, an audio spatialization effect in three-dimensional space can be established to provide users with a stereo audio-visual effect.

[0008] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments, but is not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram of an embodiment of a video and audio processing device connected to a video and audio capture device and a video and audio playback device according to the present invention;

[0010] Figure 2 A block diagram of another embodiment of the video and audio processing device connected to the video and audio capturing device and the video and audio playing device according to the present invention;

[0011] Figure 3 is a flow chart of an embodiment of the video and audio processing method according to the present invention;

[0012] Figure 4 A schematic diagram of an embodiment of a display layout;

[0013] Figure 5 A schematic diagram of another embodiment of a display panel;

[0014] Figure 6 A schematic diagram of another embodiment of a display panel;

[0015] Figure 7for Figure 6 A schematic diagram of a display screen presented on a display panel.

[0016] Among them, the reference numerals

[0017] 1: Audio and video processing device

[0018] 10: Input unit

[0019] 11: Image processing unit

[0020] 12: Sound processing unit

[0021] 13: Output port

[0022] 2,21,22: Video and audio capture device

[0023] 3: Audio and video playback device

[0024] 31: Left layout position

[0025] 32: Right page position

[0026] 33: Top layout position

[0027] 34: Bottom layout position

[0028] 35: Front page position

[0029] 36: Rear layout position

[0030] 41: Target

[0031] 42: Target

[0032] V1, V2: Initial video

[0033] Vo: Output video signal

[0034] A1, A2: audio signal

[0035] C1: Control signal

[0036] L1: Layout information

[0037] L2: Preset reference point

[0038] L3, L4: length information

[0039] W1, W2: width information

[0040] S1, S2: Stereo audio signal

[0041] So: Integrate audio signals

[0042] O1, O2: origin coordinate information

[0043] P1, P2: center point coordinate information

[0044] D1,D2: Depth distance

[0045] S01~S04: Steps DETAILED DESCRIPTION

[0046] The structural principle and working principle of the present invention are described in detail below in conjunction with the accompanying drawings:

[0047] Please refer to Figure 1 or Figure 2 In some embodiments, the video processing device 1 can be connected between the video capture device 2 and the video playback device 3. Although the figure shows two video capture devices 2 (such as video capture devices 21 and 22) and two video playback devices 3, the video capture devices 2 and the video playback devices 3 of the present invention are not limited to this number.

[0048] Taking the audio and video processing device 1 coupled to two audio and video capture devices 21 and 22 as an example, the audio and video capture device 21 can shoot the target 41 to generate a video V1, and the video V1 includes a dynamic image of the target 41. The dynamic image of the target 41 is composed of multiple first image frames at a first frame rate. In addition, the audio and video capture device 22 can shoot the target 42 to generate a video V2, and the video V2 includes a dynamic image of the target 42. The dynamic image of the target 42 is composed of multiple second image frames at a second frame rate. Here, the first frame rate and the second frame rate can be the same or different. When the audio and video capture devices 21 and 22 capture the initial videos V1 and V2, the audio and video capture devices 21 and 22 can also synchronously capture the sound of the target 41 and the sound of the target 42, and generate an audio signal A1 of the target 41 and an audio signal A2 of the target 42.

[0049] In one embodiment, the initial videos V1 and V2 captured by the video capture devices 21 and 22 may be a single video stream (ie, forming one video file) or may be two or more video streams (ie, forming two video files).

[0050] Reference Figures 1 to 3, the audio-visual processing device 1 can receive initial videos V1, V2 and audio signals A1, A2 of different targets 41, 42 from different audio-visual capture devices 21, 22 (step S01). After receiving the initial videos V1, V2 and audio signals A1, A2, the audio-visual processing device 1 can generate at least one output video signal Vo according to the received initial videos V1, V2 (step S02), and adjust each audio signal A1, A2 to a stereo audio signal S1, S2 according to the layout configuration information L1 (step S03). Then, the audio-visual processing device 1 outputs the output video signal Vo and the stereo audio signal S1, S2 to one or more audio-visual playback devices 3 (step S04), and each audio-visual playback device 3 can play the output video signal Vo, and the display screen of each audio-visual playback device 3 corresponds to the initial videos V1, V2 on the display block corresponding to the layout configuration information L1.

[0051] Here, the layout configuration information L1 may be the configuration state of the display layout of the output screen, that is, the number of display blocks included in the output screen and the orientation and / or size of each display block. In the output screen, each display block can be used to display the image frame of the original video of the corresponding single image source. Among them, the output screen may refer to the display screen displayed by the audio and video playback device 3. In some embodiments, the output screen may also refer to the display screen displayed by the built-in display screen (not shown) of the audio and video processing device 1.

[0052] Furthermore, the audio-visual processing device 1 can specify different configuration positions in the display layout of the audio-visual playback device 3 where the initial videos V1 and V2 are displayed. In some embodiments, the layout configuration information L1 includes configuration positions Lo1 and Lo2 corresponding to the initial videos V1 and V2 from different sources. Among them, the configuration positions Lo1 and Lo2 in the layout configuration information L1 correspond one-to-one to the display blocks included in the output screen, and each configuration position Lo1 and Lo2 defines the layout position for the corresponding output block. In some embodiments, each configuration position Lo1 and Lo2 corresponds to an image source. For example, in the layout configuration information L1, the image source can be represented by a source number, and each configuration position Lo1 and Lo2 can be indexed by the corresponding source number.

[0053] In some embodiments, please refer to Figure 1 or Figure 2 The video and audio processing device 1 comprises an input unit 10, an image processing unit 11, a sound processing unit 12 and an output port 13. The input unit 10 is coupled to the image processing unit 11 and the sound processing unit 12. The image processing unit 11 is coupled to the sound processing unit 12. The output port 13 is coupled to the image processing unit 11 and the sound processing unit 12.

[0054] The input unit 10 is coupled to one or more video and audio capture devices 21, 22, and the output port 13 is coupled to one or more video and audio playback devices 3. The input unit 10 can receive initial videos V1, V2 and audio signals A1, A2 from the video and audio capture devices 21, 22, and then further provide the received initial videos V1, V2 to the image processing unit 11, and provide the received audio signals A1, A2 to the sound processing unit 12.

[0055] Here, the image processing unit 11 can also receive the layout information L1. In one embodiment, the layout information L1 can be obtained by generating a control signal C1 for setting the layout by operating a user interface (not shown) of the video and audio processing device 1 by the user.

[0056] In one embodiment, the storage unit may store a plurality of preset layout configuration information in advance, and the layout configuration information of these preset preset layouts is used to indicate the configuration status (e.g., the number of sub-pictures, the image source of each sub-picture, the display position of each sub-picture, the size of each sub-picture or any combination thereof) of the display layouts of different display screens (e.g., picture-in-picture, parallel screen, floating screen, stacked screen, etc.). The user generates a selection signal of a specific display layout to be used by operating the user interface (e.g., a combination of a touch screen or a display screen and a physical button, etc.), and then the image processing unit 11 reads the corresponding preset layout configuration information from the storage unit according to the selection signal as the layout configuration information L1. In another embodiment, the storage unit may store a preset layout configuration information in advance, which is the configuration status of the display layout (i.e., the preset layout) of the display screen displayed by the preset audio and video playback device 3. When started, the image processing unit 11 may read the preset layout configuration information from the storage unit as the layout configuration information L1. Here, when the user generates a setting signal by operating the user interface, the image processing unit 11 obtains the corresponding layout information according to the setting signal to update the currently used layout information L1.

[0057] In some embodiments, the image processing unit 11 may set the layout positions of the initial videos V1 and V2 displayed on the video playback device 3 according to the layout information L1 .

[0058] For example, two image sources of initial videos V1 and V2 are provided respectively, and the display layout indicated by the layout configuration information L1 has two sub-screen layout positions. In one embodiment, the display layout can be a layout state where the screens are arranged side by side, that is, the display screen is divided into two left and right sub-screens. In the display layout, if Figure 4As shown, the layout position of the initial video V1 and the layout position of the initial video V2 may be a first position located on the left and a second position located on the right relative to the preset reference point L2, respectively. The preset reference point L2 may be the center point of the display layout. For example, the entire layout is divided into two left and right blocks of the XY plane. The first position may be a left layout position 31 (i.e., a left half block) located on the left relative to the preset reference point L2 in the XY plane, and the second position may be a right layout position 32 (i.e., a right half block) located on the right relative to the preset reference point L2 in the XY plane. At this time, the image processing unit 11 may set the initial video V1 of the first image source to be displayed at the left layout position 31 and set the initial video V2 of the second image source to be displayed at the right layout position 32 according to the layout configuration information L1. In another embodiment, the display layout may be in a configuration state of stacked screens, that is, the display screen is divided into two upper and lower sub-screens. In the display layout, if Figure 5 As shown, the layout position of the initial video V1 and the layout position of the initial video V2 may be a first position located above and a second position located below relative to the preset reference point L2, respectively. The preset reference point L2 may be the center point of the display layout. For example, the entire layout is divided into two upper and lower blocks in the XY plane, the first position may be an upper layout position 33 located above relative to the preset reference point L2 in the XY plane (i.e., the upper half block), and the second position may be a lower layout position 34 located below relative to the preset reference point L2 in the XY plane (i.e., the lower half block). At this time, the image processing unit 11 may set the initial video V1 of the first image source to be displayed at the upper layout position 33 according to the layout configuration information L1, and may set the initial video V2 of the second image source to be displayed at the lower layout position 34 according to the layout configuration information L1.

[0059] In another embodiment, the display layout may be in a picture-in-picture configuration state, that is, the display screen is divided into two sub-screens, one large and one small. In the display layout, the layout position of the initial video V1 and the layout position of the initial video V2 may be respectively a first position with a larger layout and a second position with a smaller layout and overlapping with the first position (in terms of front view). At this time, the image processing unit 11 may set the initial video V1 of the first image source to be displayed at the first position according to the layout configuration information L1, and may set the initial video V2 of the second image source to be displayed at the second position according to the layout configuration information L1.

[0060] Reference Figures 1 to 3 The sound processing unit 12 receives the layout information L1 from the image processing unit 11, so that the sound processing unit 12 can adjust the audio signals A1 and A2 to stereo audio signals S1 and S2 according to the layout positions Lo1 and Lo2 corresponding to the initial videos V1 and V2 in the layout information L1 (step S03).

[0061] In some embodiments, the number of output video signals Vo can be multiple, and they correspond one-to-one to the initial videos V1 and V2. Here, each output video signal Vo is composed of the corresponding initial video V1 or V2. At this time, the output port 13 can also output stereo audio signals S1 and S2 in the form of multiple audio streams, such as Figure 1 shown.

[0062] In the first embodiment of step S02, the image processing unit 11 can add a source tag to the data stream of the initial videos V1 and V2 to form an output video signal Vo, so as to use the source tag carried by each initial video V1 and V2 to inform the audio-visual playback device 3 to display the layout position of each initial video V1 and V2. Among them, the source tag can correspond to or be the same as the data representing the image source in the currently used layout configuration information L1. In other words, the image processing unit 11 can integrate each initial video V1 and V2 and its corresponding configuration position Lo1 and Lo2 into a stream (i.e., the output video signal Vo), and then output the output video signal Vo to the audio-visual playback device 3. In this way, when receiving the output video signal Vo, the audio-visual playback device 3 can display the image frame of the output video signal Vo at the corresponding display position in the display screen, that is, the positions of the display screen corresponding to the configuration positions Lo1 and Lo2 respectively present the image frames of the initial videos V1 and V2.

[0063] In the second embodiment of step S02, the image processing unit 11 can associate the source tags of the initial videos V1 and V2 with the data representing the image source in the layout configuration information L1, so as to use the source tags carried by each initial video V1 and V2 to inform the audio-visual playback device 3 of the layout position of the display. For example, before transmitting the output video signal Vo, the image processing unit 11 first transmits the association information to the audio-visual playback device 3, and then transmits the output video signal Vo to the audio-visual playback device 3. The association information records the corresponding relationship between the source tag and the layout position Lo1 and Lo2 of the layout configuration information L1. The audio-visual playback device 3 then learns the layout position Lo1 and Lo2 of the initial video V1 and V2 according to the source tag of each initial video V1 and V2, the received association information and the pre-stored currently used layout configuration information L1, and displays the initial video V1 and V2 at the corresponding layout position in the display screen according to the configuration position Lo1 and Lo2.

[0064] In the third embodiment of step S02, the image processing unit 11 can add data indicating the configuration positions Lo1 and Lo2 of the initial videos V1 and V2 to the data streams of the initial videos V1 and V2, so that the video playback device 3 can use the data indicating the configuration positions Lo1 and Lo2 to know the layout positions of the initial videos V1 and V2. For example, before transmitting each initial video V1 and V2, the image processing unit 11 obtains each configuration position Lo1 and Lo2 from the layout information L1, and outputs them to the two channels of the output port 13 to transmit the initial videos V1 and V2 respectively. Specifically, the image processing unit 11 first outputs the data indicating the configuration positions Lo1 and Lo2 of the initial videos V1 and V2 to the video playback device 3 through the channels corresponding to the initial videos V1 and V2, and then continues to output the initial videos V1 and V2 to the video playback device 3 through the channels.

[0065] In some other embodiments, the number of output video signals Vo may be a single one, and the output video signal Vo is composed of the initial videos V1 and V2. For example, the output video signal Vo may be an integrated video spliced ​​by the initial videos V1 and V2. In this case, the output port 13 may output the stereo audio signals S1 and S2 in the form of a single audio stream, that is, output a single stereo audio stream (hereinafter referred to as the integrated audio signal So) by integrating the stereo audio signals S1 and S2.

[0066] In the fourth embodiment of step S02, the image processing unit 11 may first assign the layout positions of the initial videos V1 and V2 to the layout positions Lo1 and Lo2 in the layout configuration information L1 in a one-to-one manner, and then, upon receiving the initial videos V1 and V2, splice the initial videos V1 and V2 into a single output video signal Vo (e.g., Figure 3 Specifically, the image processing unit 11 combines or cuts and then combines the first image frame (i.e., the initial video V1) and the second image frame (i.e., the initial video V2) at the same playback time point according to the corresponding configuration positions Lo1 and Lo2 in the layout configuration information L1 to form a third image frame (i.e., the output video signal Vo), and then outputs the third image frame to the video playback device 3 via the output port 13. Part of the third image frame is the first image frame (complete screen) or a part of the first image frame, and another part of the third image frame is the second image frame (complete screen) or a part of the second image frame.

[0067] Furthermore, in step S03, after adjusting the audio signals A1 and A2 into stereo audio signals S1 and S2 according to the layout information L1, the sound processing unit 12 also synthesizes the stereo audio signals S1 and S2 into a single stereo audio stream (hereinafter referred to as the integrated audio signal So) according to the playing time, and then outputs the integrated audio signal So to each audio and video playback device 3 via the output port 13. Upon receiving the output video signal Vo and the integrated audio and video signal So, the audio and video playback device 3 will synchronously play the output video signal Vo and the integrated audio signal So.

[0068] Then, refer to Figure 1 (or Figure 2 )and Figure 3 The sound processing unit 12 outputs one or more output video signals Vo and stereo audio signals S1 and S2 (i.e., individual independent stereo audio signals S1 and S2 or integrated audio signals So after the stereo audio signals S1 and S2 are integrated) composed of the initial videos V1 and V2 to the audio and video playback device 3 via the output port 13 (step S04), and the audio and video playback device 3 can synchronously play the initial videos V1 and V2 and the stereo audio signals S1 and S2 of the targets 41 and 42. Here, the playback effect of the stereo audio signal S1 corresponds to the layout position of the initial video V1 displayed on the audio and video playback device 3. The playback effect of the stereo audio signal S2 corresponds to the layout position of the initial video V2 displayed on the audio and video playback device 3.

[0069] In this way, the user can feel the corresponding stereo audio signals S1 and S2 through the different layout positions of the initial videos V1 and V2 according to the video playback device 3, so as to provide the user with a spatialization effect of audio.

[0070] In some embodiments, each of the configuration positions Lo1 and Lo2 may be coordinate information. In other words, the layout configuration information L1 includes coordinate information indicating the layout positions of the initial videos V1 and V2. In addition, the layout configuration information L1 further includes coordinate information of a preset reference point L2. Here, the preset reference point L2 refers to the preset position of the viewer of the output image. In some embodiments, assuming that the layout configuration of the display image is a two-dimensional configuration (such as Figure 4 and Figure 5 All the configuration positions Lo1 and Lo2 of the output screen shown are on the same XY plane), and the preset reference point L2 can be the center point of the layout configuration. In other embodiments, it is assumed that the layout configuration of the display screen is a three-dimensional configuration (such as Figure 6 The configuration positions Lo1 and Lo2 of the output screen shown are parallel to each other and located on multiple XY planes), and the preset reference point L2 can be the center point of the vertical projection formed by all layout positions in the layout configuration on the plane where the front layout position is located or on any parallel plane before the front layout position.

[0071] In some embodiments of step S03, taking two initial videos V1 and V2 as examples, the sound processing unit 12 may determine an audio adjustment value (hereinafter referred to as a first audio adjustment value) based on the coordinate information of the layout position of the initial video V1 in the layout configuration information L1 and the coordinate information of a preset reference position (such as the preset reference point L2). Furthermore, the sound processing unit 12 may determine another audio adjustment value (hereinafter referred to as a second audio adjustment value) based on the coordinate information of the layout position of the initial video V2 in the layout configuration information L1 and the coordinate information of the preset reference point L2. Then, the sound processing unit 12 adjusts the audio signal A1 to a stereo audio signal S1 according to the first audio adjustment value, and adjusts the audio signal A2 to a stereo audio signal S2 according to the second audio adjustment value.

[0072] In some embodiments of step S03, taking two initial videos V1 and V2 as examples, the sound processing unit 12 calculates the relative distance between the layout position of the initial video V1 and the preset reference point L2 based on the coordinate information corresponding to the initial video V1 and the coordinate information of the preset reference point L2, and then determines the audio adjustment value of the audio signal A1 of the initial video V1 according to the relative distance obtained above, and then uses the determined audio adjustment value to adjust the corresponding audio signal A1 to generate a stereo audio signal S1. Similarly, the sound processing unit 12 also calculates the relative distance based on the coordinate information corresponding to the initial video V2 and the coordinate information of the preset reference point L2, and then determines the audio adjustment value of the audio signal A2 of the initial video V2, thereby adjusting the audio signal A2 to generate a stereo audio signal S2.

[0073] In some embodiments, the relative distance may be proportional to the audio adjustment value. That is, if the layout position of the initial videos V1 and V2 is closer to the preset reference point L2, the audio adjustment value generated by the sound processing unit 12 is smaller, that is, the audio adjustment amount that the sound processing unit 12 needs to adjust when generating the stereo audio signals S1 and S2 corresponding to the initial videos V1 and V2 is smaller. Conversely, if the layout position of the initial videos V1 and V2 is farther away from the preset reference point L2, the audio adjustment value generated by the sound processing unit 12 is larger, and the corresponding stereo audio signals S1 and S2 need to adjust more audio adjustment amount.

[0074] In some embodiments, the coordinate information corresponding to the initial videos V1 and V2 may include designated point coordinate information and side length information. The designated point coordinate information is the position of the designated point in the display layout that defines the layout position of the initial videos V1 and V2. The designated point coordinates of the initial videos V1 and V2 can be set to any point on the layout position according to actual needs. The side length information is the side length or size that defines the layout position of the initial videos V1 and V2.

[0075] In some embodiments, assuming that the layout position of the initial videos V1 and V2 is a rectangle, the side length information may be width information W1 and W2 and length information L3 and L4. The width information W1 and W2 and the length information L3 and L4 are respectively the lengths of the two sides defining the initial videos V1 and V2 at the corresponding layout positions (e.g. Figure 4 shown).

[0076] In one embodiment, the designated point may be an origin. Taking the layout position of the initial videos V1 and V2 as a rectangle as an example, the coordinate information corresponding to the initial videos V1 and V2 may include origin coordinate information O1 and O2, width information W1 and W2, and length information L3 and L4. Among them, the origin of the layout position of the initial videos V1 and V2 may be any point on the edge of the layout position of the initial videos V1 and V2. Preferably, the origin of the layout position of the initial videos V1 and V2 may be the vertex of the layout position. Here, the width information W1 and W2 and the length information L3 and L4 may be the lengths of the two sides that define the connection origin of the layout position of the initial videos V1 and V2, respectively.

[0077] In another embodiment, the designated point may be a center point. Taking the layout position of the initial videos V1 and V2 as a rectangle as an example, the coordinate information corresponding to the initial videos V1 and V2 may include center point coordinate information P1 and P2, width information W1 and W2, and length information L3 and L4. The center point coordinate information P1 and P2 is the position of the center point of the layout position of the initial videos V1 and V2 in the display layout.

[0078] In some embodiments, when the layout of the display screen is a two-dimensional configuration (such as Figure 4 and Figure 5 ), the designated point coordinate information and the preset reference point L2 may include longitudinal data and transverse data respectively. In some embodiments, the longitudinal data and transverse data may be an X coordinate value and a Y coordinate value respectively.

[0079] In some embodiments, Figure 4 and Figure 5 As shown, the layout of the display screen is located on the XY plane. In the layout information L1, the coordinate information corresponding to the initial video V1 may include the origin coordinate information O1 of the layout position of the initial video V1 on the XY plane, the width information W1 relative to the origin coordinate information O1, and the length information L3, and the coordinate information corresponding to the initial video V2 may include the origin coordinate information O2 of the layout position of the initial video V2 on the XY plane, the width information W2 relative to the origin coordinate information O2, and the length information L4. For example, referring to Figure 4, taking the layout configuration of the display screen as dividing the entire layout into the left layout position 31 and the right layout position 32 as an example. The origin coordinate information O1 and the origin coordinate information O2 may be the coordinates of the vertex of the left layout position 31 (i.e., the origin of the left layout position 31) and the coordinates of the vertex of the right layout position 32 (i.e., the origin of the right layout position 32), respectively. The origin coordinate information O2 is located to the right of the origin coordinate information O1. The width information W1 and the length information L3 are respectively the lengths of the two sides connected by the origin on the left layout position 31. The width information W2 and the length information L4 are respectively the lengths of the two sides connected by the origin on the right layout position 32. Taking the layout configuration of the display screen as dividing the entire layout into the upper layout position 33 and the lower layout position 34 as an example. The origin coordinate information O1 and the origin coordinate information O2 may be the coordinates of the vertex of the upper layout position 33 (i.e., the origin of the upper layout position 33) and the coordinates of the vertex of the lower layout position 34 (i.e., the origin of the lower layout position 34), respectively, and the origin coordinate information O2 is located below the origin coordinate information O1. The width information W1 and the length information L3 are respectively the lengths of the two sides connected by the origin on the upper layout position 33. The width information W2 and the length information L4 are respectively the lengths of the two sides connected by the origin on the lower layout position 34. It should be clear that although the vertex of the display block of each sub-screen is used as the origin of this layout position as an example, it is not limited to this. In fact, the same relative position (i.e., any point) of the display block of each sub-screen can be arbitrarily selected as the origin of this layout position. Preferably, the origin of each layout position can be any vertex of the display block.

[0080] In some embodiments of step S03, the sound processing unit 12 may calculate the center point coordinate information P1, P2 of the layout position of the initial videos V1, V2 according to the origin coordinate information O1, O2, width information W1, W2 and length information L3, L4 of the initial videos V1, V2. Next, the sound processing unit 12 calculates the relative distance between the layout position of the initial videos V1, V2 and the preset reference point L2 (i.e., the absolute distance between the center point and the preset reference point L2) according to the center point coordinate information P1, P2 and the coordinate information of the preset reference point L2. Then, the sound processing unit 12 determines the audio adjustment value of the initial videos V1, V2 according to the relative distance obtained above, and uses the determined audio adjustment value to adjust the corresponding audio signals A1, A2 to generate stereo audio signals S1, S2.

[0081] In one embodiment, the sound processing unit 12 may substitute the origin coordinate information O1, the width information W1 and the length information L3 into the center point calculation formula to calculate the center point coordinate information P1 of the layout position of the initial video V1 (eg, Figure 4The X coordinate value and Y coordinate value of the center point of the left layout position 31 shown in the figure). Next, the sound processing unit 12 substitutes the center point coordinate information P1 and the coordinate information of the preset reference point L2 into the distance calculation formula to calculate the relative distance between the layout position of the initial video V1 and the preset reference point L2 on the XY plane (hereinafter referred to as the first distance). Then, the sound processing unit 12 determines the first audio adjustment value based on the first distance. Similarly, the sound processing unit 12 can substitute the origin coordinate information O2, the width information W2 and the length information L4 into the center point calculation formula to calculate the center point coordinate information P2 of the layout position of the initial video V2 (such as, Figure 4 Then, the sound processing unit 12 substitutes the center point coordinate information P2 and the coordinate information of the preset reference point L2 into the distance calculation formula to calculate the relative distance between the layout position of the initial video V2 and the preset reference point L2 on the XY plane (hereinafter referred to as the second distance). Then, the sound processing unit 12 determines the second audio adjustment value according to the second distance.

[0082] When the coordinate information already contains the center point coordinate information P1 and P2, in some embodiments of step S03, the sound processing unit 12 can directly calculate the relative distance between the layout position of the initial video V1 and V2 and the preset reference point L2 according to the center point coordinate information P1 and P2 and the coordinate information of the preset reference point L2. Then, the sound processing unit 12 determines the audio adjustment value of the initial video V1 and V2 according to the relative distance obtained above, and uses the determined audio adjustment value to adjust the corresponding audio signals A1 and A2 to generate a stereo audio signal S1 (S2).

[0083] In some embodiments, the sound processing unit 12 may not need to actually calculate the relative distance between the layout position of each initial video V1, V2 and the preset reference point L2, but instead obtain the pre-stored audio adjustment value by comparing the configuration position Lo1, Lo2 of each initial video V1, V2 with the preset reference point L2.

[0084] In some embodiments of step S03, the sound processing unit 12 can compare the configuration positions Lo1 and Lo2 of the initial videos V1 and V2 with the coordinate information of the preset reference point L2 (e.g., comparing the origin coordinate information O1 and O2 with the coordinate information of the preset reference point L2) to determine whether the layout position of the initial videos V1 and V2 is the first position or the second position, and obtain the audio adjustment value corresponding to the judgment result from the pre-stored audio adjustment values, and then adjust the corresponding audio signals A1 and A2 with the obtained audio adjustment value to generate stereo audio signals S1 and S2.

[0085] Taking two initial videos V1 and V2 as examples, when the sound processing unit 12 determines that the layout position of the initial video V1 is the first position and the layout position of the initial video V2 is the second position, the sound processing unit 12 can determine the first audio adjustment value according to the preset correspondence between the first position and the audio adjustment value, and determine the second audio adjustment value according to the preset correspondence between the second position and the audio adjustment value. For example, the storage unit stores a correspondence table of various layout positions and their corresponding audio adjustment values. The sound processing unit 12 searches the correspondence table for the same layout position with the first position and the second position. Then, the sound processing unit 12 reads out the audio adjustment value corresponding to the layout position that is the same as the first position from the storage unit as the first audio adjustment value, and reads out the audio adjustment value corresponding to the layout position that is the same as the second position from the storage unit as the second audio adjustment value. On the contrary, when the sound processing unit 12 determines that the layout position of the initial video V1 is the second position and the layout position of the initial video V2 is the first position, the sound processing unit 12 can determine the first audio adjustment value according to the preset corresponding relationship between the second position and the audio adjustment value, and determine the second audio adjustment value according to the preset corresponding relationship between the first position and the audio adjustment value. In this way, the sound processing unit 12 can save the amount of calculation of the actual relative distance value.

[0086] For example, taking the first position as Figure 4 The left layout position shown is 31 and the second position is Figure 4 Take the right layout position 32 shown as an example. The sound processing unit 12 compares the horizontal data of the origin coordinate information O1 and O2 corresponding to the initial videos V1 and V2 with the horizontal data of the coordinate information of the preset reference point L2. When the horizontal data of the origin coordinate information O1 and O2 is greater than the horizontal data of the coordinate information of the preset reference point L2, the sound processing unit 12 determines that the layout position of the initial videos V1 and V2 is located to the right relative to the preset reference point L2, that is, the layout position of the initial videos V1 and V2 is determined to be the right layout position 32. On the contrary, when the horizontal data of the origin coordinate information O1 and O2 is not greater than the horizontal data of the coordinate information of the preset reference point L2, the sound processing unit 12 determines that the layout position of the initial videos V1 and V2 is located to the left relative to the preset reference point L2, that is, the layout position of the initial videos V1 and V2 is determined to be the left layout position 31.

[0087] In another example, the first position is Figure 5 The upper layout position shown is 33 and the second position is Figure 5Take the lower layout position 34 shown as an example. The sound processing unit 12 compares the longitudinal data of the origin coordinate information O1 and O2 corresponding to the initial videos V1 and V2 with the longitudinal data of the coordinate information of the preset reference point L2. When the longitudinal data of the origin coordinate information O1 is greater than the longitudinal data of the coordinate information of the preset reference point L2, the sound processing unit 12 determines that the layout position of the initial videos V1 and V2 is above the preset reference point L2, that is, the layout position of the initial videos V1 and V2 is determined to be the upper layout position 33. On the contrary, when the longitudinal data of the origin coordinate information O2 is not greater than the longitudinal data of the coordinate information of the preset reference point L2, the sound processing unit 12 determines that the layout position of the initial video V2 is below the preset reference point L2, that is, the layout position of the initial video V2 is determined to be the lower layout position 34.

[0088] In some embodiments, each audio adjustment value includes adjustment information of at least one signal parameter of the corresponding audio signal A1 (or A2). The adjustment information may be, for example, volume information, delay information, frequency information, or phase information. The volume information is used to adjust the amplitude of the audio signal A1 (or A2). The delay information is used to adjust the delay time of the audio signal A1 (or A2). The frequency information is used to adjust the frequency component of the audio signal A1 (or A2). The phase information is used to adjust the phase of the audio signal A1 (or A2).

[0089] In some embodiments, the audio and video playback device 3 includes at least two channels (e.g., a left channel and a right channel) and the layout position of the initial video V1 is the left layout position 31 and the layout position of the initial video V2 is the right layout position 32. To establish a stereo audio signal S1 with a stereo sound effect relative to the X-axis direction, the first audio adjustment value may include volume information and / or delay information. In the embodiment of step S03, the sound processing unit 12 may determine the volume information and / or delay information of the first audio adjustment value according to the coordinate information of the left layout position 31 (i.e., the origin coordinate information O1, the length information L3 and the width information W1). Next, the sound processing unit 12 may attenuate the volume of the audio signal A1 of the initial video V1 according to the volume information of the first audio adjustment value (e.g., reduce the amplitude of the audio signal A1 by a specific value or a specific ratio according to the volume information), and add a delay time to the audio signal A1 according to the delay information of the first audio adjustment value, so as to generate a stereo audio signal S1 including a first signal component and a second signal component. The attenuated and delayed audio signal A1 becomes the first signal component of the stereo audio signal S1, and the unattenuated and undelayed audio signal A1 becomes the second signal component of the stereo audio signal S1. Then, the sound processing unit 12 sends the stereo audio signal S1 including the first signal component and the second signal component to the audio and video playback device 3. The audio and video playback device 3 plays the first signal component of the stereo audio signal S1 through its right channel, and the audio and video playback device 3 plays the second signal component of the stereo audio signal S1 through its left channel.

[0090] Similarly, in order to establish a stereo audio signal S2 with a stereo sound effect relative to the X-axis direction, the second audio adjustment value may include volume information and / or delay information. In one embodiment of step S03, the sound processing unit 12 may determine the volume information and / or delay information of the second audio adjustment value according to the coordinate information of the right layout position 32 (i.e., the origin coordinate information O2, the length information L4 and the width information W2), and the sound processing unit 12 may attenuate the volume of the audio signal A2 of the initial video V2 according to the volume information of the second audio adjustment value, and the sound processing unit 12 may further add a delay time to the audio signal A2 according to the delay information of the second audio adjustment value, so as to generate a stereo audio signal S2 including a first signal component and a second signal component. The attenuated and delayed audio signal A2 becomes the first signal component of the stereo audio signal S2, and the unattenuated and undelayed audio signal A2 becomes the second signal component of the stereo audio signal S2. Then, the sound processing unit 12 sends the stereo audio signal S2 including the first signal component and the second signal component to the audio and video playback device 3. The audio / video playback device 3 plays the first signal component of the stereo audio signal S2 through its left channel, and the audio / video playback device 3 plays the second signal component of the stereo audio signal S2 through its right channel.

[0091] Therefore, the user of the video playback device 3 can feel that the main sound of the initial video V1 comes from the left, and feel that the main sound of the initial video V2 comes from the right. In other words, the stereo audio signal S1 of the initial video V1 can correspond to the layout position of the initial video V1 in the video playback device 3 (i.e., the left layout position 31), and the stereo audio signal S2 of the initial video V2 can correspond to the layout position of the initial video V2 in the video playback device 3 (i.e., the right layout position 32).

[0092] In other words, the sound processing unit 12 can also adjust the volume (ie, amplitude) and / or delay time of the audio signals A1, A2 of the initial videos V1, V2 according to the left and right positions of the designated points corresponding to the initial videos V1, V2 relative to the preset reference point L2.

[0093] In some embodiments, taking the aforementioned audio and video playback device 3 including at least two channels (hereinafter referred to as the first channel and the second channel) and the layout position of the initial video V1 being the upper layout position 33 and the layout position of the initial video V2 being the lower layout position 34 as an example, in order to establish a stereo audio signal S1 with a stereo sound effect relative to the Y-axis direction, the aforementioned first audio adjustment value may include volume information and / or frequency information. In the embodiment of step S03, the sound processing unit 12 may determine the volume information and / or frequency information of the first audio adjustment value according to the coordinate information of the upper layout position 33 (e.g., the origin coordinate information O1, the length information L3, and the width information W1). Then, the sound processing unit 12 may attenuate the volume of the audio signal A1 of the initial video V1 according to the volume information of the first audio adjustment value, and the sound processing unit 12 may further adjust the frequency component of the audio signal A1 according to the frequency information of the first audio adjustment value by means of equalization (EQ) (e.g., the sound processing unit 12 may suppress the high frequency component of the audio signal A1), thereby generating a stereo audio signal S1 including a first signal component and a second signal component. Here, the audio signal A1 after attenuation and adjustment of the frequency component becomes the first signal component of the stereo audio signal S1, and the audio signal A1 without attenuation and adjustment of the frequency component may become the second signal component of the stereo audio signal S1. Then, the sound processing unit 12 sends the stereo audio signal S1 including the first signal component and the second signal component to the audio and video playback device 3. The audio / video playback device 3 plays the first signal component of the stereo audio signal S1 through its first channel, and the audio / video playback device 3 plays the second signal component of the stereo audio signal S1 through its second channel.

[0094] Similarly, in order to establish a stereo audio signal S2 with a stereo sound effect relative to the Y-axis direction, the second audio adjustment value may include volume information and / or frequency information. In the embodiment of step S03, the sound processing unit 12 may determine the volume information and / or frequency information of the second audio adjustment value according to the coordinate information of the lower layout position 34 (e.g., the origin coordinate information O2, the length information L4, and the width information W2). Then, the sound processing unit 12 may attenuate the volume of the audio signal A2 of the initial video V2 according to the volume information of the second audio adjustment value, and the sound processing unit 12 may further adjust the frequency component of the audio signal A2 according to the frequency information of the second audio adjustment value by equalization means, so as to generate a stereo audio signal S2 including a first signal component and a second signal component. The audio signal A2 after attenuation and adjustment of the frequency component becomes the first signal component of the stereo audio signal S2, and the audio signal A2 without attenuation and adjustment of the frequency component may become the second signal component of the stereo audio signal S2. The sound processing unit 12 then sends the stereo audio signal S2 including the first signal component and the second signal component to the video playback device 3. The video playback device 3 plays the first signal component of the stereo audio signal S2 with its second channel, and plays the second signal component of the stereo audio signal S2 with its first channel.

[0095] Therefore, the user of the video playback device 3 can feel that the main sound of the initial video V1 comes from above, and feel that the main sound of the initial video V2 comes from below. In other words, the stereo audio signal S1 of the initial video V1 can correspond to the layout position of the initial video V1 in the video playback device 3 (i.e., the upper layout position 33), and the stereo audio signal S2 of the initial video V2 can correspond to the layout position of the initial video V2 in the video playback device 3 (i.e., the lower layout position 34).

[0096] In other words, the sound processing unit 12 can also adjust the volume and / or frequency components of the audio signal A1 (A2) of the initial video V1 (V2) according to the upper and lower positions of the designated points corresponding to the initial videos V1 and V2 relative to the preset reference point L2.

[0097] In some embodiments, when the layout of the display screen is a three-dimensional configuration (such as Figure 6 ), the coordinate information of the designated point and the coordinate information of the preset reference point L2 may include longitudinal data, transverse data and depth data. In some embodiments, the longitudinal data, transverse data and depth data may be X coordinate values, Y coordinate values ​​and Z coordinate values ​​respectively.

[0098] In some embodiments of step S03, the sound processing unit 12 may also calculate the depth distance according to the depth data in the coordinate information of the designated point corresponding to each initial video V1, V2 and the depth data in the coordinate information of the preset reference point L2, and determine the phase information in the audio adjustment value of the audio signal A1, A2 of the initial video V1, V2 according to the calculated depth distance. In other embodiments, the sound processing unit 12 may determine the layout position of the initial video V1, V2 by comparing the coordinate information of the designated point corresponding to the initial video V1, V2 with the coordinate information of the preset reference point L2, and then obtain the audio adjustment value (including the phase information) corresponding to the determination result from the preset multiple audio adjustment values ​​(including the phase information).

[0099] In some examples, the layout of the display screen is located on the XYZ plane (eg Figure 6 ) and the display screen is a picture-in-picture screen (as shown Figure 7 Please refer to Figure 6 and Figure 7 , the layout positions of the initial videos V1 and V2 may be respectively the front layout position 35 and the rear layout position 36 with different depth distances D1 and D2 on the Z axis relative to the preset reference L2. The layout position of the initial video V1 may be the front layout position 35, and the layout position of the initial video V2 may be the rear layout position 36.

[0100] In the embodiment of step S03, the sound processing unit 12 can also calculate the depth distance D1 according to the depth data in the center point coordinate information P1 of the front layout position 35 and the depth data in the coordinate information of the preset reference point L2, and the sound processing unit 12 determines the phase information in the first audio adjustment value according to the calculated depth distance D1 (e.g., the phase information can be proportional to the depth distance D1). Similarly, the sound processing unit 12 can also calculate the depth distance D2 according to the center point coordinate information P2 of the rear layout position 36 and the coordinate information of the preset reference point L2, and the sound processing unit 12 determines the phase information in the second audio adjustment value according to the calculated depth distance D2 (e.g., the phase information can be proportional to the depth distance D2).

[0101] For example, the aforementioned audio-visual playback device 3 includes a first channel and a second channel, and the layout position of the initial video V1 is the front layout position 35 and the layout position of the initial video V2 is the rear layout position 36. To establish a stereo audio signal S1 with a stereo sound effect relative to the Z-axis direction, the aforementioned first audio adjustment value may include volume information and phase information. In the embodiment of step S03, the sound processing unit 12 may determine the volume information and phase information of the first audio adjustment value according to the coordinate information of the front layout position 35 (e.g., the origin coordinate information O1, the length information L3 and the width information W1). Then, the sound processing unit 12 may attenuate the volume of the audio signal A1 of the initial video V1 according to the volume information of the first audio adjustment value, and the sound processing unit 12 may further adjust the phase of the audio signal A1 according to the phase information of the first audio adjustment value (e.g., the sound processing unit 12 may adjust the phase of the audio signal A1 in an inverted manner), thereby generating a stereo audio signal S1 including the first signal component and the second signal component. The attenuated and phase-adjusted audio signal A1 becomes the first signal component of the stereo audio signal S1, and the unattenuated and phase-unadjusted audio signal A1 becomes the second signal component of the stereo audio signal S1. Then, the audio processing unit 12 sends the stereo audio signal S1 including the first signal component and the second signal component to the audio and video playback device 3. The audio and video playback device 3 plays the first signal component of the stereo audio signal S1 with its first channel, and the audio and video playback device 3 plays the second signal component of the stereo audio signal S1 with its second channel.

[0102] Similarly, in order to establish a stereo audio signal S2 with a stereo sound effect relative to the Z-axis direction, the aforementioned second audio adjustment value may include volume information and phase information. In the embodiment of step S03, the sound processing unit 12 may determine the volume information and phase information of the second audio adjustment value according to the coordinate information of the rear layout position 36 (e.g., the origin coordinate information O2, the length information L4, and the width information W2). Then, the sound processing unit 12 may attenuate the volume of the audio signal A2 of the initial video V2 according to the volume information of the second audio adjustment value, and the sound processing unit 12 may further adjust the phase of the audio signal A2 according to the phase information of the second audio adjustment value (e.g., the sound processing unit 12 may adjust the phase of the audio signal A2 in an inverted manner), thereby generating a stereo audio signal S2 including a first signal component and a second signal component. The attenuated and phase-adjusted audio signal A2 becomes the first signal component of the stereo audio signal S2, and the unattenuated and phase-unadjusted audio signal A2 becomes the second signal component of the stereo audio signal S2. The sound processing unit 12 then sends the stereo audio signal S2 including the first signal component and the second signal component to the audio-visual playback device 3. The audio-visual playback device 3 plays the first signal component of the stereo audio signal S2 with its second channel, and the audio-visual playback device 3 plays the second signal component of the stereo audio signal S2 with its first channel. Therefore, the user of the audio-visual playback device 3 can feel that the main sounds of the initial videos V1 and V2 located at different depths of the layout have different distances.

[0103] In another example, the first position is Figure 6 The front layout position 35 and the second position are shown Figure 6Take the rear layout position 36 shown and the front layout position 35 and the rear layout position 36 as an example, both of which are located behind the preset reference point L2. The sound processing unit 12 can compare the depth data of the origin coordinate information O1 and O2 corresponding to the initial videos V1 and V2 with the depth data of the coordinate information of the preset reference point L2 to obtain the layout positions of the initial videos V1 and V2. When the depth data corresponding to the initial video V1 is less than the depth data of the coordinate information of the preset reference point L2 and the depth data corresponding to the initial video V2, the sound processing unit 12 determines that the layout position of the initial video V1 is at the rear relative to the preset reference point L2, that is, the layout position of the initial video V1 is determined to be the rear layout position 36, and then obtains the audio adjustment value corresponding to the rear layout position 36 from the preset multiple audio adjustment values. On the contrary, when the depth data corresponding to the initial video V1 is smaller than the depth data of the coordinate information of the preset reference point L2 but larger than the depth data corresponding to the initial video V2, the sound processing unit 12 determines that the layout position of the initial video V1 is located behind the preset reference point L2 and in front of the layout position of the initial video V2, that is, the layout position of the initial video V1 is determined to be the front layout position 35, and then the audio adjustment value corresponding to the front layout position 35 is obtained from the preset multiple audio adjustment values.

[0104] In other words, the sound processing unit 12 may further adjust the volume and / or phase of the audio signals A1 and A2 of the initial videos V1 and V2 according to the depth of the designated point corresponding to the initial videos V1 and V2 relative to the preset reference point L2.

[0105] In one embodiment, the first channel may be a left channel, and the second channel may be a right channel. In another embodiment, the first channel may be an upper channel, and the second channel may be a lower channel.

[0106] Here, the above embodiments are merely examples of how to adjust an audio signal to a stereo audio signal, and the adjustments are made to the volume, delay, phase, frequency or other arbitrary combinations of sound parameters to make the stereo audio signal more suitable for current usage. For example, by adjusting two, three or more sound parameters such as the volume, delay and phase of the audio signal to obtain a stereo audio signal, the user can hear the sound coming from one of the left front, right front, left rear, right rear, upper left, upper right, lower left or lower right directions. In other words, one or more sound parameters can be adjusted as needed to give the listener a better sense of stereo.

[0107] It should be clear that in the above embodiments, although each video capture device 2 outputs the initial video V1 (or V2) and the audio signal A1 (or A2) to the video processing device 1, the invention is not limited thereto. In other words, in some embodiments, the video capture device 2 can output a single video stream to the video processing device 1, and then the video separation unit (not shown) of the video processing device 1 can separate the video and audio signals from the input video stream. The video separation unit is coupled between the input unit 10 and the image processing unit 11 and between the input unit 10 and the sound processing unit 12.

[0108] Similarly, in some embodiments, the video processing device 1 can output a single video stream to the video playback device 3. Here, the video processing device 1 can combine the output video signal Vo and the integrated audio signal So of the initial video V1 and V2 corresponding to the source mark or configuration position Lo1 and Lo2 into a single video stream through the video synthesis unit (not shown), and then output it to the video playback device 3 through the output port 13. Here, the video synthesis unit is coupled between the image processing unit 11 and the output port 13 and between the sound processing unit 12 and the output port 13.

[0109] In some embodiments, the aforementioned image processing unit 11 and sound processing unit 12 (as well as the video and audio synthesis unit and / or the video and audio synthesis unit) can be implemented by one or more processing units. Each processing unit can be any analog and / or digital device based on an operation instruction operation signal, but is not limited thereto.

[0110] In some embodiments, the aforementioned storage unit may be coupled to the image processing unit 11 or built into the image processing unit 11. Here, the storage unit may be implemented by one or more storage elements, wherein the storage element may be, for example, a memory or a register, but is not limited thereto.

[0111] In some embodiments, the input unit 10 can be implemented by a plurality of connection ports or by one or more network modules. Each connection port can be coupled to an external device (i.e., the video capture device 2) of the video processing device 1 through a transmission line. Here, the connection port supports video, image, or audio transmission. The network module can comply with any wireless communication protocol or wired communication protocol.

[0112] In some embodiments, the output port 13 may be a physical audio / video connector, or a connector supporting any wireless communication protocol.

[0113] In some embodiments, when the video processing device 1 is coupled to multiple video playback devices 3 , all video playback devices 3 can receive the same file from the video processing device 1 , ie, receive the same output video signal Vo and stereo audio signals S1 , S2 (or So) respectively.

[0114] In some embodiments, the audio spatialization audio and video processing method according to the present invention can be implemented by a computer program product, so that when the audio and video processing device 1 loads the program and executes it, the audio spatialization audio and video processing method of any embodiment can be completed. In other words, the audio spatialization audio and video processing method of any embodiment is implemented on the audio and video processing device 1 by one or more processing units executing firmware or software algorithms. In some embodiments, the computer program product can be a readable recording medium, and the above program is stored in the readable recording medium for loading by the audio and video processing device 1. In some embodiments, the above program itself can be a computer program product, and is transmitted to the audio and video processing device 1 via wired or wireless means.

[0115] In summary, according to any embodiment of the audio-visual processing method and the audio-visual processing device of the present invention, the audio-visual processing device 1 can adjust the signal parameters of the audio signal of the video according to the display position of the video. In some embodiments, the stereo audio signals S1, S2 (or So) have differences in volume, delay, phase, frequency or other combinations thereof compared with the audio signals A1, A2; or the stereo audio signals S1, S2 (or So) correspond to the channels of the audio-visual playback device 3, and here, when the stereo audio signals S1, S2 (or So) are output on different channels, they are different from each other in the signal parameters of volume, delay, phase, frequency or other combinations thereof. In this way, an audio spatialization effect in three-dimensional space can be established to provide users with a stereo audio-visual effect.

[0116] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. An audio-visual processing method for audio spatialization, applicable to an audio-visual processing device, the audio-visual processing device is connected to an audio-visual playback device, characterized in that: The video and audio processing method comprises: Receive a plurality of initial videos and an audio signal corresponding to each of the initial videos through the video and audio processing device; Generate at least one output video signal according to the plurality of initial videos by the video processing device; The audio and video processing device adjusts each audio signal into a stereo audio signal according to a layout configuration information; and Outputting the at least one output video signal and each of the stereo audio signals to the video playback device through the video processing device, so that the video playback device outputs a display screen having the plurality of initial videos displayed on the corresponding plurality of display blocks respectively according to the layout configuration information by playing the at least one output video signal; The at least one output video signal includes the plurality of initial videos and information or marks indicating the configuration positions of the initial videos in the display panel of the video playback device, and the video playback device learns the configuration positions of the initial videos in the display screen according to the information or marks and displays the initial videos in the display blocks at the corresponding configuration positions. The layout configuration information includes coordinate information corresponding to each of the initial videos, and the step of adjusting each of the audio signals into each of the stereo audio signals according to the layout configuration information by the video and audio processing device includes: Determine the phase information in the audio adjustment value according to the depth data in the coordinate information corresponding to each of the initial videos and the depth data in the coordinate information of the preset reference point; and The phase of the corresponding audio signal is adjusted according to the phase information.

2. The audio-visual processing method of audio spatialization according to claim 1, characterized in that: The number of the at least one output video signal is one, a display layout of the audio and video playback device includes multiple layout positions corresponding to the multiple display blocks respectively, and the step of generating the at least one output video signal according to the multiple initial videos by the audio and video processing device includes: configuring the multiple initial videos at the multiple layout positions according to the layout configuration information by the audio and video processing device to form the output video signal.

3. The audio-visual processing method of audio spatialization according to claim 1, characterized in that: The at least one output video signal includes the multiple initial videos, a display layout of the audio and video playback device includes multiple layout positions corresponding to the multiple display blocks respectively, and the step of the audio and video playback device outputting the display screen having the multiple initial videos on the multiple display blocks corresponding to the layout configuration information by playing the at least one output video signal includes: displaying the multiple initial videos at the multiple layout positions according to the layout configuration information by the audio and video playback device.

4. The audio-visual processing method of audio spatialization according to claim 1, characterized in that: The step of adjusting each of the audio signals into each of the stereo audio signals according to the layout information by the audio and video processing device further includes: Determine the frequency information in the audio adjustment value according to the longitudinal data in the coordinate information of each of the initial videos and the longitudinal data in the coordinate information of the preset reference point; Adjust the frequency component of the corresponding audio signal according to the frequency information; Determine the delay information of the audio adjustment value according to the horizontal data in the coordinate information of each of the initial videos and the horizontal data in the coordinate information of the preset reference point; and The delay time of the corresponding audio signal is adjusted according to the delay information.

5. The audio-visual processing method of audio spatialization according to claim 1, characterized in that: The step of determining the phase information in the audio adjustment value according to the depth data in the coordinate information corresponding to each of the initial videos and the depth data in the coordinate information of the preset reference point comprises: Determine a center point of a layout position corresponding to one of the plurality of display blocks according to each of the coordinate information; and The audio adjustment value is determined according to a relative distance between the center point and the preset reference point.

6. The audio spatialization audio and video processing method according to claim 1, characterized in that: The number of the plurality of display blocks is two, a display panel displayed by the video playback device includes a first position and a second position respectively corresponding to the plurality of display blocks and relative to the preset reference point, and the step of determining the phase information in the audio adjustment value according to the depth data in the coordinate information corresponding to each of the initial videos and the depth data in the coordinate information of the preset reference point includes: Determining, according to the coordinate information and the preset reference point, whether a layout position of the initial video is the first position or the second position; and The audio adjustment value is determined according to a determination result that the layout position is the first position or the second position.

7. The audio-visual processing method of audio spatialization according to claim 1, characterized in that: The preset reference point refers to the preset position of the viewer of the display screen, and the layout configuration information further includes the side length information corresponding to each of the initial videos. The coordinate information corresponding to each of the initial videos is the coordinate information of a specified point defining a layout position of the initial video in a display layout of the audio and video playback device, and the side length information defines the size of the layout position.

8. An audio and video processing device, suitable for connecting to an audio and video playback device, characterized in that: The video and audio processing device comprises: an image processing unit, receiving a plurality of original videos and generating at least one output video signal according to the plurality of original videos; a sound processing unit connected to the image processing unit, receiving a layout configuration information and an audio signal corresponding to each of the initial videos, and adjusting each of the audio signals into a stereo audio signal according to the layout configuration information; and an output port coupled to the image processing unit and the sound processing unit, outputting the stereo audio signal and the at least one output video signal to the video playback device, so that the video playback device outputs a display screen having the plurality of initial videos displayed on the corresponding plurality of display blocks respectively according to the layout configuration information by playing the at least one output video signal; The at least one output video signal includes the multiple initial videos and information or marks indicating the configuration positions of the initial videos in the display panel of the video playback device, and the video playback device learns the configuration positions of the multiple initial videos in the display screen according to the information or marks and displays the multiple initial videos on the display blocks at the corresponding configuration positions; and Among them, the layout configuration information includes coordinate information corresponding to each of the initial videos, and in adjusting each of the audio signals to the stereo audio signals, the sound processing unit determines the phase information in the audio adjustment value based on the depth data in the coordinate information corresponding to each of the initial videos and the depth data in the coordinate information of the preset reference point and adjusts the phase of the corresponding audio signal according to the phase information.

9. The video and audio processing device according to claim 8, characterized in that: The number of the at least one output video signal is one, a display layout displayed by the audio-visual playback device includes multiple layout positions corresponding to the multiple display blocks respectively, and the image processing unit configures the multiple initial videos on the multiple layout positions according to the layout configuration information to form the output video signal.

10. The video and audio processing device according to claim 8, characterized in that: In adjusting each of the audio signals to the stereo audio signal, the sound processing unit also determines the frequency information of the audio adjustment value according to the longitudinal data in each of the coordinate information and the longitudinal data in the coordinate information of the preset reference point, adjusts the corresponding frequency component of the audio signal according to the frequency information, determines the delay information of the audio adjustment value according to the horizontal data in each of the coordinate information and the horizontal data in the coordinate information of the preset reference point, and generates the stereo audio signal according to the delay time of the audio signal corresponding to the adjustment.

11. The video and audio processing device according to claim 8, characterized in that: The preset reference point refers to the preset position of the viewer of the display screen, and the layout configuration information further includes the side length information corresponding to each of the initial videos. The coordinate information corresponding to each of the initial videos defines the coordinate information of a specified point of a layout position in a display layout of the audio and video playback device, and the side length information defines the size of the layout position.

Images