Method and device for splicing display of multi-path video and storage medium

Abstract

The application discloses a multi-path video splicing display method and device and a storage medium. The multi-path video splicing display method comprises the following steps: splicing a plurality of video streams and displaying the spliced video, obtaining a current video display progress in the display process, the video display progress is used for indicating the position of a current playing frame in the entire spliced video; based on the video display progress, the number of selected video streams and / or the picture fusion weight are adjusted to obtain an adjusted number and / or an adjusted weight; based on the adjusted number, a corresponding number of video streams are selected for splicing processing, and / or based on the adjusted weight, the plurality of video streams are spliced to generate a next playing frame. In the display process of the spliced video, the number of selected video streams and / or the picture fusion weight can be flexibly adjusted according to the current video display progress, the multi-path video stream is flexibly spliced, various video display effects are obtained, and the user watching experience is improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a method, device and storage medium for splicing and displaying multiple video streams. Background Technology

[0002] With the development of visual imaging technology and video acquisition technology, video acquisition has evolved from capturing videos from ordinary perspectives to wide-angle videos, and further to 360-degree and even 720-degree panoramic videos. To acquire panoramic videos, a panoramic camera consisting of multiple lenses is needed to capture videos from multiple directions, and then the videos captured by the multiple lenses from multiple directions are stitched together to form a panoramic video.

[0003] Currently, multiple video streams are generally spliced ​​together for display, resulting in a limited display method and a poor user viewing experience. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides at least one method, device, and storage medium for splicing and displaying multiple video streams.

[0005] The first aspect of this application provides a method for splicing and displaying multiple video streams. The method includes: splicing multiple video streams and displaying the spliced ​​video; obtaining the current video display progress during the display process, wherein the video display progress is used to indicate the position of the current playback frame in the entire spliced ​​video; adjusting the number of selected video streams and / or the image fusion weight based on the video display progress to obtain the adjusted number and / or the adjusted weight; selecting a corresponding number of video streams for splicing based on the adjusted number, and / or splicing multiple video streams based on the adjusted weight to generate the next playback frame.

[0006] In one embodiment, the display progress of the spliced ​​video is controlled in the time dimension, with different playback frames corresponding to different times, and / or, the display progress of the spliced ​​video is controlled in the spatial dimension, with different playback frames corresponding to different observation point positions. The video display progress includes time progress parameters and / or spatial progress parameters. During the display process, the current video display progress is obtained, including: obtaining the time information corresponding to the current playback frame to obtain the time progress parameters; and / or obtaining the spatial position of the observation point corresponding to the current playback frame to obtain the spatial progress parameters.

[0007] In one embodiment, the video display progress includes a time progress parameter; based on the video display progress, the number of selected video streams and / or the image blending weight are adjusted to obtain the adjusted number and / or adjusted weight, including: determining the time interval in which the time progress parameter is located, obtaining the current time interval, querying the number of video streams matching the current time interval, and obtaining the adjusted number of video streams; and / or, each video stream has a preset reference time point, and the adjusted weight of each video stream is determined based on the relationship between the time progress parameter and the reference time point of the video stream.

[0008] In one embodiment, the video display progress includes a spatial progress parameter; based on the video display progress, the number of selected video streams and / or the image blending weight are adjusted to obtain the adjusted number and / or adjusted weight, including: determining the spatial interval where the spatial progress parameter is located, obtaining the current spatial interval, querying the number of video streams matching the current spatial interval, and obtaining the adjusted number of video streams; and / or, each video stream has a preset reference position point, and the image blending weight of each video stream is determined based on the positional relationship between the spatial progress parameter and the reference position point of the video stream.

[0009] In one embodiment, a corresponding number of video streams are selected based on the adjusted quantity and spliced ​​together to obtain the next playback frame. This includes: obtaining the spatial position of the image in each video stream to obtain the video position; obtaining the spatial position of the observation point corresponding to the current playback frame to obtain the current observation position; selecting the corresponding number of video streams whose video positions are closest to the current observation position based on the selection quantity to obtain the target video stream; and splicing each target video stream to obtain the next playback frame.

[0010] In one embodiment, the method further includes: determining the initial spatial position of the observation point of the video frame to obtain the initial observation position; selecting an initial video stream and / or setting an initial fusion weight based on the initial observation position; performing splicing processing on the initial video stream and / or splicing processing on multiple video streams based on the initial fusion weight to generate an initial display frame.

[0011] In one embodiment, determining the initial spatial position of the observation point in the video frame to obtain the initial observation position includes: scoring the image quality of each video stream and / or calculating the image overlap between each video stream; selecting video streams whose image quality scores meet preset scoring conditions and / or whose image overlap meets preset overlap conditions to obtain preferred video streams; and determining the initial spatial position of the observation point based on the spatial position corresponding to the image in the preferred video stream to obtain the initial observation position.

[0012] In one embodiment, selecting an initial video stream and / or setting an initial fusion weight based on the initial observation position includes: calculating the distance between the initial observation position and the spatial position corresponding to the image in each video stream to obtain the point-to-point spacing for each video stream; selecting video streams whose point-to-point spacing meets a preset distance condition to obtain an initial video stream; or, setting an initial fusion weight for each video stream based on the point-to-point spacing, wherein the point-to-point spacing and the initial fusion weight are inversely correlated; or, selecting video streams whose point-to-point spacing meets a preset distance condition to obtain an initial video stream, and setting an initial fusion weight for each initial video stream based on the point-to-point spacing.

[0013] A second aspect of this application provides a multi-channel video splicing and display device, comprising: a progress acquisition module, used to splice multiple video streams and display the spliced ​​video, acquiring the current video display progress during the display process, the video display progress being used to indicate the position of the current playback frame in the entire spliced ​​video; an adjustment module, used to adjust the number of selected video streams and / or the image fusion weight based on the video display progress, to obtain the adjusted number and / or the adjusted weight; and a splicing module, used to select a corresponding number of video streams for splicing based on the adjusted number, and / or to splice multiple video streams based on the adjusted weight, to generate the next playback frame.

[0014] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described method for splicing and displaying multiple video streams.

[0015] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described method for splicing and displaying multiple video streams.

[0016] The above solution involves splicing multiple video streams and displaying the spliced ​​video. During the display process, the current video display progress is obtained, indicating the position of the current playback frame within the entire spliced ​​video. Based on the video display progress, the number of selected video streams and / or the image blending weights are adjusted to obtain the adjusted number and / or adjusted weights. A corresponding number of video streams are selected for splicing based on the adjusted number, and / or multiple video streams are spliced ​​based on the adjusted weights to generate the next playback frame. This allows for flexible adjustment of the number of selected video streams and / or image blending weights during the spliced ​​video display process, enabling flexible splicing of multiple video streams, resulting in diverse video display effects and improving the user viewing experience.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 This is a schematic diagram illustrating the implementation environment of the solution in an exemplary embodiment of this application; Figure 2 This is a flowchart illustrating a multi-channel video splicing and display method according to an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating the adjustment of the number of video streams selected, as shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating the aerial dive video effect of an exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating multipath progressive fusion during the movement of the observation point, as shown in an exemplary embodiment of this application; Figure 6 This is a block diagram illustrating a multi-channel video splicing display device according to an exemplary embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application; Figure 8 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0020] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0021] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0022] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0023] The following describes the method for splicing and displaying multiple video streams provided in the embodiments of this application.

[0024] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an implementation environment of the scheme according to an exemplary embodiment of this application. The implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are interconnected.

[0025] The number of terminals 110 can be one or more. Terminal 110 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0026] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0027] In one example, server 120 can splice multiple video streams to obtain a spliced ​​video. Then, server 120 can store the spliced ​​video locally, send it back to terminal 110, or transmit it to other terminals.

[0028] In one example, a client application running a target application is installed on terminal 110. This target application could be an application that provides video stitching functionality, and the stitched video is obtained by stitching multiple video streams together using this target application. Server 120 can be a backend server for this target application, used to provide background services to the client of the target application.

[0029] The multi-channel video splicing display method provided in this application embodiment can be executed by a terminal 110, such as a client of the target application installed and running on the terminal 110, or by a server 120, or by the interaction and cooperation between the terminal 110 and the server 120, that is, some steps of the method are executed by the terminal 110 and other steps are executed by the server 120.

[0030] Of course, besides Figure 1 In addition to the implementation environment shown, this application can also be applied to other implementation environments and executed by other electronic devices in other implementation environments, and this application does not limit it.

[0031] It is understood that in the specific embodiments of this application, data such as videos, images, and user information are involved. When the various embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0032] Please see Figure 2 , Figure 2 This is a flowchart illustrating a multi-channel video splicing and display method according to an exemplary embodiment of this application, as shown below. Figure 2 As shown, the method for splicing and displaying multiple video streams includes at least steps S210 to S230, which are detailed below: Step S210: Perform splicing processing on multiple video streams and display the spliced ​​video. During the display process, obtain the current video display progress, which is used to indicate the position of the current playback frame in the entire spliced ​​video.

[0033] Multiple video streams can be stitched together to create a spliced ​​video with a wider field of view than a single video stream.

[0034] Among them, multiple video streams can be pre-stored in memory, and multiple video streams can be read from memory and stitched together; multiple video streams can also be acquired in real time, such as multiple video acquisition devices capturing images of a specified scene from different directions, generating video streams from different directions, and receiving the video streams uploaded by each video acquisition device and stitching them together.

[0035] Optionally, the video stream can be preprocessed before splicing. Preprocessing methods include, but are not limited to, distortion correction, noise removal, and brightness adjustment. This application does not limit the scope of such preprocessing.

[0036] This can be achieved by detecting matching image features between different video streams, and then performing image registration on each video stream based on these matching features to obtain a stitched video, ensuring continuity at the stitching point. The matching image features refer to image features with a similarity greater than a preset similarity threshold.

[0037] Of course, other image stitching methods can also be used, such as pre-marking the connection points between video streams and stitching them directly according to the connection points. This application does not limit the specific method of video stream stitching.

[0038] Display the spliced ​​video.

[0039] If the spliced ​​video is displayed on a connected local display device, the playback frames of the spliced ​​video can be sent directly to the local display device. If it needs to be forwarded to other display devices for display, in order to reduce the amount of data transmitted over the network, the spliced ​​video needs to be encoded to obtain an encoded bitstream. The display device then decodes the received encoded bitstream and displays it.

[0040] During the display of the spliced ​​video, the current video display progress is obtained, which is used to indicate the position of the current playback frame in the entire spliced ​​video.

[0041] Step S220: Based on the video display progress, adjust the number of selected video streams and / or the image blending weight to obtain the adjusted number and / or adjusted weight.

[0042] Based on the current video display progress, flexibly adjust the number of video streams selected and / or the image blending weights used to generate the spliced ​​video to obtain the adjusted number and / or adjusted weights.

[0043] The selected quantity limits the total number of video streams used to generate the spliced ​​video.

[0044] The number of selected streams controls the field of view of the final stitched video. The more selected streams, the more video streams are used for stitching, resulting in a stitched video with a wider field of view. The fewer selected streams, the fewer video streams are used for stitching, resulting in a stitched video with a narrower field of view.

[0045] The image blending weight is used to limit the weight used by each video stream when performing video stream splicing processing.

[0046] The image blending weight controls the degree to which the image content of the corresponding video stream is displayed in the final stitched video. The higher the image blending weight of the video stream, the more image content is displayed in the video stream, and the lower the image blending weight of the video stream, the less image content is displayed in the video stream.

[0047] For example, the size of the displayable image area of ​​the video stream can be controlled by the image blending weight. The larger the image blending weight, the larger the displayable image area. In order to ensure the continuity of the video frame, the image content at the video stream splicing point is displayed first. Alternatively, the transparency of the image content of the video stream can be controlled by the image blending weight. The larger the image blending weight, the lower the transparency. In order to ensure the continuity of the video frame, the image content at the video stream splicing point can have a smooth transition in transparency.

[0048] It should be noted that the selection quantity and / or image blending weight can be adjusted every time a new playback frame needs to be generated; or the selection quantity and / or image blending weight can be adjusted every preset period; or the selection quantity and / or image blending weight can be adjusted when a preset event is detected (such as detecting that any video stream contains a target object). This application does not limit the triggering conditions for the adjustment.

[0049] Step S230: Select a corresponding number of video streams for splicing based on the adjusted quantity, and / or splice multiple video streams based on the adjusted weights to generate the next playback frame.

[0050] Based on the adjusted quantity and / or adjusted weight corresponding to the current video display progress, generate a display screen that adapts to the current video display progress.

[0051] For example, the number of video streams can be adjusted based on the video display progress. Specifically, a corresponding number of video streams are selected based on the adjusted number and spliced ​​together to generate the next playback frame.

[0052] The stitching process can be as follows: it can be done by directly registering images of each video stream based on the matched image features, without involving image fusion weights; it can also be done by directly using preset image fusion weights to weight and stitch each video stream; or it can be done by calculating the image quality scores of each video stream to be stitched, determining the image fusion weight of each video stream based on the image quality scores, and the image quality scores are positively correlated with the image fusion weights, so that the video streams can be weighted and stitched according to the image fusion weights of each video stream.

[0053] For example, the frame blending weights of the video streams can be adjusted only based on the video display progress to obtain the adjusted weights. Specifically, multiple video streams are spliced ​​together based on the adjusted weights to generate the next playback frame.

[0054] Each video stream can have an initial frame blending weight pre-set, which can then be adjusted based on the video display progress (e.g., increasing, decreasing, or adding / reducing specific values). Alternatively, the frame blending weight of each video stream in the previous adjustment period can be obtained and adjusted based on the current video display progress. Another approach is to pre-set a weight calculation function that defines the mapping relationship between different video display progresses and different frame blending weights, substituting the current video display progress into the weight calculation function to obtain the adjusted weight.

[0055] For example, the number of video streams and their image blending weights can be adjusted simultaneously based on the video display progress. Specifically, a corresponding number of video streams are selected based on the adjusted number, and each selected video stream is spliced ​​together according to its adjusted weight to generate the next playback frame.

[0056] Taking a 4-channel video stream as an example, four video acquisition devices capture images of a specified scene from four directions, generating video streams in four directions, denoted as Video Stream 1, Video Stream 2, Video Stream 3, and Video Stream 4. Responding to the panoramic video display command sent by the video display terminal, the number of video streams selected and / or the image blending weights are controlled at different video display progresses to generate the opening video corresponding to the panoramic video, which is then sent to the video display terminal for playback.

[0057] During the opening video playback (e.g., during time t∈[0.0, 1.0]), the number of selected video streams and / or the image fusion weights are dynamically adjusted based on the scheme of this application.

[0058] For example, please see Figure 3 , Figure 3 This is a schematic diagram illustrating the adjustment of the number of video streams selected, as shown in an exemplary embodiment of this application. Figure 3 As shown, the number of video streams selected gradually increases as the video progresses in the opening video. Taking controlling the video display progress in the time dimension as an example, the opening video generation process includes: at time t∈[0.0, 0.2], a display screen is generated based on video stream 1. At this time, the screen is presented as a single-channel fisheye or wide-angle image with a narrow field of view, serving as the starting point for the opening; at t∈[0.2, 0.6], video stream 2, which has matching image features with video stream 1, is added, and a display screen is generated based on video stream 1 and video stream 2, expanding the field of view to approximately 180°; at t∈[0.6, 0.8], video stream 3, which has matching image features with video stream 2, is added, and a display screen is generated based on video stream 1, video stream 2, and video stream 3, expanding the field of view to approximately 270°; at 0.8≥t, video stream 4, which has matching image features with video stream 3, is added, and a display screen is generated based on video stream 1, video stream 2, video stream 3, and video stream 4, completing a full 720° spherical panoramic stitching (at this time, the horizontal field of view is approximately 360°), fully opening the field of view and generating a complete opening video. The above embodiments can gradually increase the screen coverage during the video opening process, realizing a gradual visual progression from single-channel to multi-channel, thereby enhancing the user's immersive viewing experience.

[0059] Of course, in addition to the above-mentioned methods of adjusting the selection quantity and / or image blending weight, other methods can also be used to adjust the selection quantity and / or image blending weight of the video stream. For example, an exit video that is the opposite of the opening video can be generated, in which the selection quantity of the video stream is reduced as the video display progresses. This application does not limit this.

[0060] By flexibly adjusting the number of selected video streams and / or the image blending weights according to the current video display progress during the video splicing process, the next playback frame of the spliced ​​video can be obtained, realizing flexible splicing of multiple video streams and improving the user viewing experience.

[0061] The following describes some embodiments of this application in detail.

[0062] In some implementations, the initial display screen in the task of generating and displaying spliced ​​video can be generated according to fixed rules, such as selecting a preset number of video streams to splice together to obtain the initial display screen. Of course, the initial display screen can also be flexibly generated according to the actual application scenario. Here, the initial display screen refers to the display screen at the beginning of the spliced ​​video, which is at the starting point of the video display progress.

[0063] For example, the initial spatial position of the observation point in the video frame is determined to obtain the initial observation position; based on the initial observation position, an initial video stream is selected and / or an initial fusion weight is set; the initial video stream is spliced, and / or multiple video streams are spliced ​​based on the initial fusion weight to generate the initial display image.

[0064] The initial spatial position of the observation point can be preset, such as the position of looking directly at video stream 1 as the initial position of the observation point by default; the initial spatial position of the observation point can also be selected by the user, such as displaying an observation point selection interface to the user, which displays all spatial positions that can be used as observation points, and obtaining the initial position of the observation point in response to the user's selection command.

[0065] Of course, the initial position of the observation point can also be flexibly determined according to the actual application scenario.

[0066] For example, each video stream is scored for picture quality, and / or the overlap between each video stream is calculated; video streams whose picture quality scores meet preset scoring conditions and / or whose overlap meets preset overlap conditions are selected to obtain preferred video streams; based on the spatial positions of the images in the preferred video streams, the initial spatial positions of the observation points are determined to obtain the initial observation positions.

[0067] Picture quality rating is used to reflect the information quality corresponding to the image content in a video stream. It can be based on the clarity of the image content in the video stream, and / or the number of target objects, and / or the type of target objects, and / or the presence of key events, and / or the degree of image occlusion, etc. The higher the clarity, the more target objects, and / or the key target objects, and / or the presence of key events, the higher the picture quality rating.

[0068] Image overlap is used to reflect the similarity of image content between different video streams. It can be calculated by counting the number of matching image features between different video streams. The more matching image features, the higher the image overlap. It can also be calculated by counting the area of ​​images belonging to the same region between different video streams. The larger the area of ​​images belonging to the same region, the higher the image overlap.

[0069] Select the best video from multiple video streams based on image quality score and / or image overlap.

[0070] Specifically, video streams that meet preset scoring conditions for image quality and / or preset overlap conditions for image overlap are selected to obtain the preferred video stream.

[0071] Preset scoring conditions include, but are not limited to, a picture quality score greater than a preset scoring threshold, or the highest picture quality score, or the picture quality score being among the top N after being sorted in descending order.

[0072] The preset overlap conditions include, but are not limited to, the image overlap being greater than the preset overlap threshold, or the image overlap being the largest, or the image overlap being ranked in the top N after being sorted in descending order.

[0073] If the number of preferred video streams is zero, the user can be prompted to select an initial observation position, or the default initial observation position can be used.

[0074] If there is only one preferred video stream, the spatial position corresponding to the image in the preferred video stream can be directly used as the initial observation position. Alternatively, the initial observation position can be obtained by summing the spatial position corresponding to the image in the preferred video stream with preset position change parameters (such as forward or backward).

[0075] If there are multiple preferred video streams, the spatial position corresponding to the image in any preferred video stream can be used as the initial observation position, or the center point of the spatial position corresponding to the image in multiple preferred video streams can be calculated to obtain the initial observation position.

[0076] The spatial location corresponding to the image in the video stream refers to the physical location corresponding to the image content. It can be obtained by converting the image coordinates of the reference points (such as the center point, corner point, or landmark point) contained in the video stream image into world coordinates through a pre-calibrated coordinate transformation matrix. Alternatively, it can be obtained by acquiring the deployment location of the video acquisition device corresponding to the video stream and directly using the deployment location of the video acquisition device as the spatial location corresponding to the image in the video stream. This application does not limit this.

[0077] For example, to label the spatial positions of multiple cameras: establish a coordinate system relative to the origin and obtain the position (xi, yi, zi) of each camera in the coordinate system.

[0078] Based on the initial observation location, select the initial video stream and / or set the initial fusion weights.

[0079] For example, the distance between the initial observation position and the corresponding spatial position in each video stream is calculated to obtain the point spacing for each video stream; the initial video stream is selected and / or the initial fusion weight is set based on the point spacing.

[0080] For example, select a video stream whose point spacing meets the preset distance condition to obtain the initial video stream.

[0081] The preset distance conditions include, but are not limited to, minimum distance between points, distance between points less than a preset distance threshold, or the points ranked in the top M after ascending order of distance.

[0082] For example, the initial fusion weights for each video stream are set based on the pixel spacing, and the pixel spacing and the initial fusion weights are inversely correlated.

[0083] That is, the smaller the distance between points, the greater the initial fusion weight, and the greater the distance between points, the smaller the initial fusion weight.

[0084] For example, select video streams whose point spacing meets the preset distance condition to obtain initial video streams, and set the initial fusion weights corresponding to each initial video stream based on the point spacing.

[0085] In addition to selecting an initial video stream that meets the preset distance conditions, you can also set the initial fusion weight for each initial video stream based on the distance between the points. Similarly, the distance between the points and the initial fusion weight are inversely correlated.

[0086] Then, the initial video stream is spliced ​​together, and / or multiple video streams are spliced ​​together based on the initial fusion weights to generate the initial display image.

[0087] The principle of "splicing the initial video stream and / or splicing multiple video streams based on the initial fusion weights" is the same as the principle of "selecting a corresponding number of video streams for splicing based on the adjusted quantity and / or splicing multiple video streams based on the adjusted weights". The specific steps of splicing will not be elaborated here.

[0088] Once the spliced ​​video starts playing, the video display progress will be updated.

[0089] The display progress of spliced ​​videos can be controlled in the time and / or spatial dimensions.

[0090] For example, the display progress of spliced ​​videos can be controlled in the time dimension. Each playback frame is sorted in chronological order, and different playback frames are displayed as time changes, that is, the playback frames at different display progresses correspond to different times.

[0091] For example, in the spatial dimension, the display progress of spliced ​​video is controlled. Each playback frame is sorted according to its spatial position, and different playback frames are displayed as the position of the observation point changes. That is, the position of the observation point is different for playback frames with different display progress.

[0092] For example, the display progress of spliced ​​videos can be controlled in the time and space dimensions. Each playback frame is sorted according to the time order and spatial position order. Different playback frames are displayed as time changes and the position of the observation point changes. That is, the time and observation point position corresponding to the playback frames with different display progress are different.

[0093] Therefore, the video display progress includes time progress parameters and / or spatial progress parameters. In step S210, obtaining the current video display progress during the display process includes: obtaining the time information corresponding to the current playback frame to obtain the time progress parameters; and / or obtaining the spatial location of the observation point corresponding to the current playback frame to obtain the spatial progress parameters.

[0094] The time progress parameter can be the system time corresponding to the current playback frame (such as 14:32:08 (the real time on the computer / mobile phone)) or the internal time of the video (such as 00:01:35 (playing to 1 minute and 35 seconds)), etc., and this application does not limit it.

[0095] The spatial progress parameter can be the coordinates of the observation point in the video image coordinate system or the coordinates in the world coordinate system, etc., and this application does not limit it.

[0096] Then, based on the video display progress, adjust the number of selected video streams and / or the image blending weights to obtain the adjusted number and / or adjusted weights.

[0097] For example, the video display progress includes a time progress parameter; in step S220, based on the video display progress, the number of selected video streams and / or the image fusion weight are adjusted to obtain the adjusted number and / or adjusted weight, including: determining the time interval in which the time progress parameter is located, obtaining the current time interval, querying the number of video streams matching the current time interval, and obtaining the adjusted number of video streams; and / or, each video stream has a preset reference time point, and the adjusted weight of each video stream is determined based on the size relationship between the time progress parameter and the reference time point of the video stream.

[0098] For example, see below. Figure 3 When the time progress parameter is in the time interval [0.0, 0.2], the number of matched video streams is 1; when the time progress parameter is in the time interval [0.2, 0.6], the number of matched video streams is 2; when the time progress parameter is in the time interval [0.6, 0.8], the number of matched video streams is 3; and when the time progress parameter is greater than 0.8, the number of matched video streams is 4.

[0099] And / or, each video stream has a preset reference time point, and the adjusted weight of each video stream is calculated based on the relationship between the time progress parameter and the reference time point of the video stream.

[0100] For example, the opening video plays during the period t∈[0.0, 1.0], with reference time points including 0.3 and 0.7. When the time progress parameter is less than 0.3, the image blending weight of video stream 1 is always 1, and the image blending weight of the other video streams is 0. When the time progress parameter is greater than 0.3, the image blending weight of video stream 1 gradually decreases, while the image blending weights of video stream 2 and video stream 3 gradually increase (wherein, the magnitude of decrease and increase can be calculated based on the time difference between the time progress parameter and the reference time point). When the time progress parameter is greater than 0.7, the image blending weights of video stream 1, video stream 2, video stream 3, and video stream 4 tend to be 0.25.

[0101] When stitching together video streams based on image blending weights, it can be expressed as the following formula 1: (Formula 1) In formula 1, The image fusion weight (adjusted weight) of the i-th video stream satisfies... , Represents the i-th video stream, Indicates azimuth. Indicates pitch angle, This represents the playback frame obtained by time t fusion.

[0102] For example, the video display progress includes a spatial progress parameter; in step S220, based on the video display progress, the number of selected video streams and / or the image fusion weight are adjusted to obtain the adjusted number and / or the adjusted weight, including: determining the spatial interval where the spatial progress parameter is located, obtaining the current spatial interval, querying the number of video streams matching the current spatial interval, and obtaining the adjusted number of video streams; and / or, each video stream has a preset reference position point, and the image fusion weight of each video stream is determined based on the positional relationship between the spatial progress parameter and the reference position point of the video stream.

[0103] The spatial interval can be one-dimensional, in which case the observation point is any point on the trajectory line, and the spatial interval corresponds to the range of the trajectory line segment; the spatial interval can also be two-dimensional, in which case the observation point is any point within the two-dimensional movable region, and the spatial interval corresponds to the range of the region divided within the two-dimensional movable region; of course, the spatial interval can also be three-dimensional, in which case the observation point is any point within the three-dimensional movable region, and the spatial interval corresponds to the range of the three-dimensional region divided within the three-dimensional movable region. This application does not limit the scope of the spatial interval.

[0104] The observation point can be moved by user operation, or it can move on its own, such as moving over time.

[0105] Based on the spatial interval where the spatial progress parameter is located, the current spatial interval is obtained. The number of video streams matching the current spatial interval is then queried to obtain the adjusted number of video streams.

[0106] For example, video streams can be stitched together to create a panoramic sphere. A smooth Bézier curve is set relative to the center of the panoramic sphere. Any coordinate on the Bézier curve can be used as an observation point. As the observation point moves downward along the Bézier curve, the visual effect of an aerial dive is simulated.

[0107] For example, please see Figure 4 , Figure 4 This is a schematic diagram illustrating the aerial dive video effect of an exemplary embodiment of this application, as shown below. Figure 4 As shown, the initial position of the observation point is at the starting point of the Bézier curve, displaying only one video stream. As the observation point descends along the Bézier curve, the number of video streams gradually increases. As the observation point approaches the center of the panoramic sphere, the video streams are gradually merged, and the field of view gradually increases. When the observation point reaches the center of the sphere, all video streams are completely merged, presenting a complete panoramic view.

[0108] And / or, each video stream has a preset reference position point, and the image fusion weight of each video stream is determined based on the positional relationship between the spatial progress parameter and the reference position point of the video stream.

[0109] Similar to the method of "determining the adjusted weight of each video stream based on the relationship between the time progress parameter and the reference time point of the video stream", the image fusion weight of each video stream is also determined based on the positional relationship between the spatial progress parameter and the reference position point of the video stream.

[0110] Continue with Figure 4 To illustrate, we establish a coordinate system with the center of the panoramic sphere as the origin. The initial spatial position of the observation point is (0, 0, R_max). At this time, the field of view of the observation point points to the center of the sphere and covers the entire scaled-down panoramic sphere.

[0111] When the observation point is at the starting point of the Bézier curve, the panoramic sphere displays only one video stream, and the panoramic sphere is scaled down according to the initial display radius R_min.

[0112] The opening video plays for a time t ∈ [0.0, 1.0]. The descent begins at t=0 and reaches the center of the sphere at t=1. The Bézier curve is represented by the following formula 2: (Formula 2) In formula 2, That is, the initial spatial position of the observation point. , For Bessel control points, The spatial location of the center of the sphere (i.e., the origin of the coordinate system). This represents the coordinates (x, y, z) of the observation point at time t.

[0113] As the observation point moves closer to the center of the sphere, the radius of the panoramic sphere gradually increases from a shrinking state to a 1:1 scale (e.g., ...). Figure 4 (The dashed lines gradually expand on both sides of the sphere). When the observation point reaches the center of the sphere, the radius of the sphere is at its maximum. The formula for the change in radius is expressed as follows: Formula 3: R(t) = R_min + t×(R_normal - R_min) (Formula 3) In Formula 3, R_normal represents the preset normal radius length, and R(t) represents the radius length at time t.

[0114] For clarity, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating multipath progressive fusion during the movement of the observation point, as shown in an exemplary embodiment of this application. Figure 5As shown, the observation point moves automatically over time t. At t=0.0, the observation point is at its initial observation position, the radius of the panoramic sphere is at its minimum, and the display screen generated using video stream 1 has a field of view of approximately 90°. As time progresses, the observation point moves, and at t=0.3, the observation point is in the middle of its dive trajectory. The radius of the panoramic sphere increases, and video stream 2, which has matching image features with video stream 1, is added. The display screen generated using video stream 1 and video stream 2 expands the field of view to approximately 180°. As time progresses further, the observation point moves even more, and at t=0... At t=7, the observation point is closer to the center of the sphere, the radius of the panoramic sphere is close to its maximum radius, and video stream 3 with matching image features with video stream 2 is added. The display screen is generated using video streams 1, 2, and 3, and the field of view is expanded to approximately 270°. As time progresses, at t=1.0, the observation point moves to the center of the sphere, the radius of the panoramic sphere is at its maximum radius, and video stream 4 with matching image features with video stream 3 is added. The display screen is generated using video streams 1, 2, 3, and 4, completing the full 720° spherical panoramic stitching.

[0115] In some implementations, a corresponding number of video streams are selected based on the adjusted quantity and spliced ​​together to obtain the next playback frame. This includes: obtaining the spatial position of the image in each video stream to obtain the video position; obtaining the spatial position of the observation point corresponding to the current playback frame to obtain the current observation position; selecting the corresponding number of video streams whose video positions are closest to the current observation position based on the selection quantity to obtain the target video stream; and splicing each target video stream to obtain the next playback frame.

[0116] In some implementations, when key image information exists in the video stream, in order to facilitate the viewing and retrieval of key image information when the user watches multi-channel spliced ​​video, the number of video streams selected and / or the image fusion weight can be flexibly adjusted according to the proximity between the video display progress and the progress point corresponding to the key image information, so as to enhance the prominence of the key image information.

[0117] Key image information includes, but is not limited to, key target objects, and / or key scenes, and / or key events. This key image information may be pre-defined or extracted from user-input text and / or images; this application does not impose any limitations on this.

[0118] Obtain the time and / or spatial location corresponding to key image information to obtain progress points.

[0119] For example, detect video frames containing key target objects in the video stream, use these video frames as key image information, determine the time corresponding to the key image information, obtain key progress points, and adjust the number of video streams selected and / or the image blending weight based on the proximity of the current video display progress to the key progress points.

[0120] For example, the information content of key image information in each video stream (such as the number of key target objects, the area of ​​key scene images, etc.) is detected separately. Video streams whose information content meets preset content conditions (such as the highest information content or information content greater than a preset content threshold) are selected as key video streams, and the remaining video streams are designated as non-key video streams. Then, the closer the current video display progress is to a key progress point, the more non-key video streams are reduced and / or the image fusion weight of more non-key video streams is reduced to highlight key image information, making it easier for users to retrieve and view key image information and improving the user viewing experience.

[0121] In some implementations, to avoid abrupt transitions during switching between multiple video streams, the image blending weights of each video stream are smoothly transitioned as the video display progresses. For example, linear interpolation or cosine interpolation is used to calculate the image blending weights at each progress point, and weight smoothing is performed.

[0122] Figure 6 This is a block diagram illustrating a multi-channel video splicing display device according to an exemplary embodiment of this application. Figure 6 As shown, the exemplary multi-channel video splicing display device 600 includes: The progress acquisition module 610 is used to splice multiple video streams and display the spliced ​​video. During the display process, it acquires the current video display progress, which is used to indicate the position of the current playback frame in the entire spliced ​​video. The adjustment module 620 is used to adjust the number of selected video streams and / or the image blending weights based on the video display progress, so as to obtain the adjusted number and / or adjusted weights. The splicing module 630 is used to select a corresponding number of video streams for splicing based on the adjusted quantity, and / or to splice multiple video streams based on the adjusted weights to generate the next playback frame.

[0123] It should be noted that the multi-channel video splicing display device and the multi-channel video splicing display method provided in the above embodiments belong to the same concept. The specific operation methods of each module and unit have been described in detail in the method embodiments and will not be repeated here. In practical applications, the multi-channel video splicing display device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation.

[0124] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating the structure of an electronic device in an exemplary embodiment of this application. The electronic device 700 includes a memory 710 and a processor 720. The processor 720 executes program instructions stored in the memory 710 to implement the steps in any of the above embodiments of the multi-channel video splicing display method. In a specific implementation scenario, the electronic device 700 may include, but is not limited to, a microcomputer or a server. Furthermore, the electronic device 700 may also include mobile devices such as laptops and tablets, without limitation.

[0125] Specifically, processor 720 controls itself and memory 710 to implement the steps in any of the above-described embodiments of the multi-channel video splicing display method. Processor 720 can also be referred to as a Central Processing Unit (CPU). Processor 720 may be an integrated circuit chip with signal processing capabilities. Processor 720 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 720 can be implemented using integrated circuit chips.

[0126] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating the structure of a computer-readable storage medium in an exemplary embodiment of this application. The computer-readable storage medium 800 stores program instructions 810 that can be executed by a processor. The program instructions 810 are used to implement the steps in any of the above embodiments of the multi-channel video splicing display method.

[0127] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0128] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for splicing and displaying multiple video streams, characterized in that, The method includes: Multiple video streams are spliced ​​together and the spliced ​​video is displayed. During the display process, the current video display progress is obtained, and the video display progress is used to indicate the position of the current playback frame in the entire spliced ​​video. Based on the video display progress, adjust the number of selected video streams and / or the image blending weight to obtain the adjusted number and / or adjusted weight; Based on the adjusted quantity, select a corresponding number of video streams for splicing processing, and / or based on the adjusted weight, splice multiple video streams to generate the next playback frame.

2. The method according to claim 1, characterized in that, The display progress of the spliced ​​video is controlled in the time dimension, with different playback frames corresponding to different times, and / or, the display progress of the spliced ​​video is controlled in the spatial dimension, with different observation point positions corresponding to playback frames. The video display progress includes time progress parameters and / or spatial progress parameters; obtaining the current video display progress during the display process includes: Obtain the time information corresponding to the current playback frame to get the time progress parameter; and / or obtain the spatial position of the observation point corresponding to the current playback frame to get the spatial progress parameter.

3. The method according to claim 2, characterized in that, The video display progress includes a time progress parameter; the step of adjusting the number of selected video streams and / or the image blending weight based on the video display progress to obtain the adjusted number and / or adjusted weight includes: Determine the time interval in which the time progress parameter is located, obtain the current time interval, query the number of video streams that match the current time interval, and obtain the adjusted number of video streams; And / or, each video stream has a preset reference time point, and the adjusted weight of each video stream is determined based on the relationship between the time progress parameter and the reference time point of the video stream.

4. The method according to claim 2, characterized in that, The video display progress includes spatial progress parameters; the step of adjusting the number of selected video streams and / or the image blending weights based on the video display progress to obtain the adjusted number and / or adjusted weights includes: Determine the spatial interval in which the spatial progress parameter is located, obtain the current spatial interval, query the number of video streams that match the current spatial interval, and obtain the adjusted number of video streams; And / or, each video stream has a preset reference position point, and the image fusion weight of each video stream is determined based on the positional relationship between the spatial progress parameter and the reference position point of the video stream.

5. The method according to claim 1, characterized in that, Based on the adjusted quantity, a corresponding number of video streams are selected and spliced ​​together to obtain the next playback frame, including: The video position is obtained by acquiring the spatial position of the frame in each video stream, and the current observation position is obtained by acquiring the spatial position of the observation point corresponding to the current playback frame. Based on the selected quantity, select a corresponding number of video streams whose video locations are closest to the current observation location to obtain the target video stream; Each target video stream is spliced ​​together to obtain the next playback frame.

6. The method according to claim 1, characterized in that, The method further includes: The initial spatial position of the observation point in the video frame is determined to obtain the initial observation position. Based on the initial observation position, select the initial video stream and / or set the initial fusion weights; The initial video stream is spliced ​​together, and / or multiple video streams are spliced ​​together based on the initial fusion weights to generate an initial display image.

7. The method according to claim 6, characterized in that, The initial spatial position of the observation point in the confirmed video frame is obtained by determining the initial observation position, including: For each video stream, score its picture quality and / or calculate the picture overlap between each video stream; The preferred video stream is obtained by selecting the video stream whose picture quality score meets the preset scoring conditions and / or whose picture overlap meets the preset overlap conditions; Based on the spatial position corresponding to the image in the preferred video stream, the initial spatial position of the observation point is determined, and the initial observation position is obtained.

8. The method according to claim 6, characterized in that, The step of selecting an initial video stream and / or setting initial fusion weights based on the initial observation position includes: Calculate the distance between the initial observation position and the corresponding spatial position of the image in each video stream to obtain the point spacing for each video stream; Select the video stream whose point spacing meets the preset distance condition to obtain the initial video stream; Alternatively, an initial fusion weight can be set for each video stream based on the point spacing, wherein the point spacing and the initial fusion weight are inversely correlated. Alternatively, select video streams whose point spacing meets the preset distance condition to obtain initial video streams, and set the initial fusion weights corresponding to each initial video stream based on the point spacing.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.

Images