Media information processing method and apparatus, and storage medium

Abstract

The application discloses a media information processing method and device and a storage medium. The media information processing method comprises the following steps: receiving a plurality of media information streams; acquiring a first display timestamp of a target media information packet received; taking the first display timestamp as a starting display timestamp of each media information stream; performing information slicing on each media information stream according to the starting display timestamp to obtain a plurality of media slice information, wherein the media slice information corresponds to a slice serial number, and all the media slice information with the same slice serial number has the same media time length; and aggregating all the target media slice information to obtain free-viewpoint media slice information, wherein the target media slice information is the media slice information with the same slice serial number. In the embodiment of the application, the user can realize seamless switching between free viewpoints, the video experience effect of the user is improved, and thus the technical blank in the related method can be made up.

Description

Technical Field

[0001] The present application relates to the field of video technology, and in particular to a media information processing method and device thereof, and a computer storage medium. Background Art

[0002] With the rapid development of 5G technology and high-speed Internet, the metaverse and true Internet are rapidly approaching, and immersive media applications are developing rapidly. The current innovative free viewpoint technology allows viewers to freely choose any 360-degree viewing angle at any time, enhancing the user's immersive experience. Users can freely switch perspectives while watching videos. However, due to the large time difference between the video streams of different perspectives shot by multiple cameras at the same time arriving at the media server, it is impossible to ensure the good picture quality of the overall picture, which greatly affects the user experience. Summary of the Invention

[0003] The embodiments of the present application provide a media information processing method and device thereof, and a computer storage medium, which can enhance the user's video experience.

[0004] In a first aspect, an embodiment of the present application provides a method for processing media information, including:

[0005] receiving a plurality of media information streams, wherein the media information streams include a plurality of media information packets;

[0006] Obtaining a first display timestamp of a received target media information packet, wherein the target media information packet is a first received media information packet among all the media information packets;

[0007] Using the first display timestamp as the start display timestamp of each of the media information streams;

[0008] Segmenting each of the media information streams according to the start display timestamp to obtain a plurality of media segment information of each of the media information streams, wherein the media segment information corresponds to a segment sequence number, and all the media segment information with the same segment sequence number have the same media duration;

[0009] The target media segment information in all the media information streams is aggregated to obtain free viewpoint media segment information, wherein the target media segment information is the media segment information having the same segment sequence number.

[0010] In a second aspect, an embodiment of the present application further provides a media information processing device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the media information processing method as described above when executing the computer program.

[0011] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the media information processing method as described above.

[0012] In an embodiment of the present application, the first display timestamp of the acquired target media information packet is uniformly set as the starting display timestamp of each media information stream to solve the defect that the images of each media information stream arrive at the media server inconsistently at the same time. In this case, the information of each media information stream is segmented according to the starting display timestamp to obtain multiple media segmentation information, and the media segmentation information with the same segmentation sequence number in all media information streams is aggregated to obtain complete free viewpoint media segmentation information, thereby ensuring the image quality while avoiding large-scale spatial jumps in the video image during the user's perspective switching process; therefore, the embodiment of the present application enables users to achieve seamless switching between free viewpoints, improves the user's video experience, and thus fills the technical gaps in related methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flow chart of a media information processing method provided by one embodiment of the present application;

[0014] Figure 2a This is a schematic diagram of multiple media information streams provided by an embodiment of the present application before alignment;

[0015] Figure 2b This is a schematic diagram of multiple media information streams after alignment provided by an embodiment of the present application;

[0016] Figure 3 This is a flowchart of obtaining multiple media fragment information of each media information stream in the media information processing method provided by another embodiment of the present application;

[0017] Figure 4 This is a flow chart before obtaining multiple media fragment information of each media information stream in the media information processing method provided by one embodiment of the present application;

[0018] Figure 5 This is a flow chart of obtaining free viewpoint media segment information in a media information processing method provided by an embodiment of the present application;

[0019] Figure 6 1 is a schematic diagram of a media server for executing a media information processing method provided by an embodiment of the present application;

[0020] Figure 7 This is a flowchart of a method for processing media information executed by an alignment module provided in one embodiment of the present application;

[0021] Figure 8 is a schematic diagram of multiple media information streams provided by another embodiment of the present application;

[0022] Figure 9 This is a flowchart of a method for processing media information executed by a splicing module provided by an embodiment of the present application;

[0023] Figure 10 This is a schematic diagram of a media information processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical methods and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0025] It should be noted that although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the flowcharts. The terms "first," "second," and the like in the specification, claims, and drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0026] At present, in order to improve the problem of large time differences when video streams from different perspectives of the same scene shot by multiple cameras reach the media server, the existing technology compresses the video streams of each camera and splices them into a large ultra-high-resolution picture, and then performs image correction. This places relatively high demands on the user's network bandwidth. As the number of cameras increases, in order to adapt to the resolution of the user's player, the resolution of the camera needs to be reduced, which greatly affects the user experience.

[0027] Based on this, the present application provides a media information processing method and its device, computer storage medium and computer program product. One embodiment of the media information processing method includes: receiving multiple media information streams, wherein the media information stream includes multiple media information packets; obtaining a first display timestamp of the received target media information packet, wherein the target media information packet is the first received media information packet among all media information packets; using the first display timestamp as the starting display timestamp of each media information stream; segmenting each media information stream according to the starting display timestamp to obtain multiple media segment information of each media information stream, wherein the media segment information corresponds to a segment sequence number, and all media segment information with the same segment sequence number has the same media duration; aggregating the target media segment information in all media information streams to obtain free viewpoint media segment information, wherein the target media segment information is the media segment information with the same segment sequence number. In this embodiment, the first display timestamp of the acquired target media information packet is uniformly set as the starting display timestamp of each media information stream to solve the defect that the images of each media information stream arrive at the media server inconsistently at the same time. In this case, each media information stream is segmented according to the starting display timestamp to obtain multiple media segmentation information, and the media segmentation information with the same segmentation sequence number in all media information streams is aggregated to obtain complete free viewpoint media segmentation information, thereby ensuring the image quality while avoiding large-scale spatial jumps in the video image during the user's perspective switching process; therefore, the embodiment of the present application enables users to achieve seamless switching between free viewpoints, improves the user's video experience, and thus fills the technical gaps in related methods.

[0028] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, Figure 1 This is a flowchart of a media information processing method provided by an embodiment of the present application. The media information processing method may include but is not limited to steps S110 to S150.

[0030] Step S110: Receive multiple media information streams, where the media information streams include multiple media information packets.

[0031] In this step, multiple media information streams are received so that the multiple received media information streams can be accurately distinguished from each other in subsequent steps, thereby determining which media information stream contains the first received media information packet.

[0032] In one embodiment, the execution entity of steps S110 to S150 and related steps can be selected and set by those skilled in the art according to specific circumstances and is not limited here. For example, a media server for coordinating and managing various media information streams is used as the execution entity, that is, multiple media information streams are received by the media server, and the following steps S120 to S150 and related steps are executed based on the multiple media information streams. Corresponding functional modules can be set in the media server to execute corresponding steps to achieve better coordination effects. Therefore, a stream receiving module can be set in the media server to pull media information streams from various camera positions at the front end of the free viewpoint and add them to the stream receiving cache queue in the stream receiving module; for example, other servers, nodes, modules or devices are set as the party that coordinates and manages the media server, that is, the multiple media information streams are indirectly processed by the coordinating and managing media server. In this case, the corresponding server, node, module or device can also serve as the execution entity of steps S110 to S150 and related steps. It should be noted that, in the following embodiments of the present application, the “media server” is mainly used as the execution entity of steps S110 to S150 and related steps for description, but it is not the only limitation.

[0033] In one embodiment, a media server is a key device in a next-generation network. Under the control of a control device (e.g., a soft switch, application server, etc.), this device provides the media resource functions required to implement various services on an IP network, including service audio provisioning, conferencing, interactive response, notifications, unified messaging, and advanced voice services. Within the application server, commands such as playback can be sent to the media server using, but are not limited to, MSML (Media Server Markup Language). The media server is highly customizable and can flexibly implement one or more functions, including but not limited to:

[0034] Dual-Tone Multi Frequency (DTMF) signal acquisition and decoding function: According to the relevant operating parameters sent by the control device, DTMF signals are received from the DTMF phone, encapsulated in the signaling and transmitted to the control device;

[0035] Sending function of recorded notification: according to the requirements of the control device, play the specified recorded notification to the user in the specified voice;

[0036] Conference function: supports audio mixing of multiple RTP streams and mixing of different encoding formats;

[0037] Conversion between different codec algorithms: supports multiple voice codec algorithms such as G.711, G.723, G.729, and can realize conversion between codec algorithms;

[0038] Automatic speech synthesis function: concatenates several speech elements or fields to form a complete voice prompt notification, which can be fixed or variable;

[0039] Dynamic voice playback / recording function: including music on hold, Follow-me voice service, etc.

[0040] Tone signal generation and transmission function: provides basic signal tones such as dial tone, busy tone, ringback tone, waiting tone and unavailable tone;

[0041] Resource maintenance and management functions: Provide maintenance and management of media resources and the equipment itself, such as data configuration and fault management, in local and / or remote ways.

[0042] The media server has at least one of the following characteristics:

[0043] Advancedness: It can adopt ITU-T H.248 and SIP standard protocols;

[0044] Compatibility: It can easily communicate with soft switch systems of different manufacturers;

[0045] High reliability: The gateway provides dual power supplies and supports hot swapping; it is positioned as a carrier-grade device with system congestion protection;

[0046] Easy to maintain: supports communication with SNMP network management, enabling online system maintenance, resource management, post-analysis, etc.

[0047] High scalability and easy upgradeability: The independent application layer can customize various value-added services for users and can update the system online to meet user needs to the greatest extent;

[0048] Flexibility: Flexible networking and powerful integrated access capabilities can provide users with a variety of solutions.

[0049] In one embodiment, there is no limitation on the reception of the media information stream of each camera position, that is, the reception method of the media information stream of different cameras can be the same, or the corresponding method can be selected for reception according to the specific setting situation, for example, the media information stream of the camera position selected in the scene is pulled in the form of Real Time Messaging Protocol (RTMP). That is to say, in the embodiment of the present application, it is sufficient to ensure that multiple media information streams can be received, and the specific reception method is not limited here. Since there is no need to limit the transmission method of the media information stream, it is also applicable to application scenarios in which the media information stream is pulled in other ways.

[0050] In one embodiment, the timing and number of receiving media information streams and media information packets in each media information stream are not limited and can be set accordingly based on specific scenarios. For example, a stadium may typically have more than 50 camera positions, corresponding to more than 50 media information streams to be received. Since users may need to enter the stadium at a specific time to watch the video, the sending time or playback time of the selected media information stream can be set near the specific time to facilitate users to watch the video at the specific time.

[0051] Step S120: Acquire a first display timestamp of the received target media information packet, wherein the target media information packet is the first received media information packet among all media information packets.

[0052] In this step, since it is necessary to solve the defect of inconsistent images of various media information streams arriving at the media server at the same time, that is, for all media information streams, regardless of the order in which they arrive at the media server, all media information streams need to be synchronized. Therefore, in order to avoid omissions or mismatches of media information streams, at least the first received media information packet needs to be found as the starting point for improvement. Therefore, the first received media information packet is found from all media information packets and used as the target media information packet. The first display timestamp of the target media information packet is obtained, so that the display timestamps of all media information packets can be aligned with the first display timestamp of the target media information packet in subsequent steps to solve the defect of inconsistent images of various media information streams arriving at the media server at the same time.

[0053] In one embodiment, the first display timestamp of the received target media information packet can be obtained in various ways, which are not limited herein. For example, the display timestamps of all media information packets can be aggregated and then compared to obtain the first display timestamp of the target media information packet.

[0054] Step S130: Using the first display timestamp as the start display timestamp of each media information stream.

[0055] In this step, by using the first display timestamp as the starting display timestamp of each media information stream, the display timestamps of each media information stream can be synchronized to the starting display timestamp, so that the display timestamps of all media information streams will be consistent, thereby solving the defect of inconsistency in that each media information stream arrives at the media server at the same time, so as to facilitate information segmentation and aggregation of each media information stream according to the starting display timestamp in subsequent steps.

[0056] A specific example is given below to illustrate the working principles and processes of the above embodiments.

[0057] Example 1:

[0058] like Figure 2a and Figure 2b As shown, Figure 2a This is a schematic diagram of multiple media information streams provided by an embodiment of the present application before alignment. Figure 2b This is a schematic diagram of multiple media information streams after alignment provided by an embodiment of the present application. As an example, a schematic diagram of the media information streams corresponding to three camera positions is given, and the media information stream in each camera position includes multiple repeated fragments.

[0059] Taking the media server as an example, after all media information packets are collected into the receiving stream cache queue, each media information packet in the receiving stream cache queue is traversed to determine whether the current media information packet is the first media information packet received. If so, the startpts of the first segment of all cameras are forcibly set, where startpts is the first presentation time stamp (PTS) of the current segment, that is, the PTS of the first media information packet of the current segment (that is, the current media information packet) received. Otherwise, the media information packet is stored in the linked list of the corresponding camera, and then the above judgment process can be repeated for another media information packet until the required first media information packet is found.

[0060] like Figure 2a As shown, a schematic diagram of the media information flow of each camera position is given without modifying the start display timestamp. The number in the box represents the PTS of the current media information packet. It can be seen that the segment length is 6s, the PTS range of the first segment of camera position 1 is [0~540000), startpts is 0, the PTS range of the first segment of camera position 2 is [7200~547200), startpts is 7200, and the PTS range of the first segment of camera position 3 is [3600~543600), startpts is 3600; since the PTS ranges of the starting segments of each camera position are inconsistent, the terminal will have a large-scale spatial jump problem of the picture when switching between camera positions.

[0061] like Figure 2bAs shown in the figure, a schematic diagram of the media information flow of each camera position when the start display timestamp is modified is given. The first media information packet received from camera position 2 (that is, the first media information packet in the receiving stream cache queue is the media information packet of camera position 2) is used as an example for explanation. The fragmentation time is 6s. It can be seen that compared with the original media information flow of the camera position, the PTS range of the first fragment of camera position 1 is [0~547200), startpts is 7200, the PTS range of the first fragment of camera position 2 is [7200~547200), startpts is 7200, and the PTS range of the first fragment of camera position 3 is [0~547200), startpts is 7200. The PTS range of a fragment is [3600~547200), and startpts is 7200. In this way, the startpts of the second fragment of each camera position is 547200. That is to say, since the startpts of the second fragment of each camera position is the same and the fragment length is also the same, then starting from the second fragment of each camera position, the subsequent fragments of each camera position can be guaranteed to be aligned respectively, so they are consistent when arriving at the media server at the same time. Therefore, the defect of inconsistent arrival of each media information stream at the media server at the same time can be solved.

[0062] Step S140: Segmenting each media information stream according to the start display timestamp to obtain multiple media segment information of each media information stream. The media segment information corresponds to a segment sequence number, and all media segment information with the same segment sequence number have the same media duration.

[0063] In this step, since the start display timestamp of each media information stream has been determined in step S130, each media information stream can be further segmented according to the start display timestamp to obtain multiple media segmentation information of each media information stream, and each media segmentation information can be distinguished by the segmentation sequence number, wherein all media segmentation information with the same segmentation sequence number has the same media duration. Therefore, for different media information streams, the media segmentation information of the same time period can be confirmed by comparing their respective segmentation sequence numbers, so that in the subsequent steps, the various media segmentation information of the same time period can be aggregated into a complete free viewpoint segment.

[0064] like Figure 3 As shown, an embodiment of the present application further illustrates step S140, which includes but is not limited to steps S141 and S142.

[0065] Step S141: For each media information stream, obtain the second display timestamp of the currently received media information packet;

[0066] Step S142: When it is determined that the information fragmentation condition is met according to the second display timestamp and the start display timestamp, initial information fragmentation is performed according to the currently received media information packet, the second display timestamp is used as the new start display timestamp, and subsequent information fragmentation is performed according to the new start display timestamp.

[0067] In this step, the second display timestamp of the currently received media information packet is obtained so that the second display timestamp can be compared with the aligned starting display timestamp to determine whether the information fragmentation condition is met. If so, initial information fragmentation can be performed based on the currently received media information packet, and subsequent information fragmentation can be performed using the second display timestamp that meets the conditions as the new starting display timestamp. In this way, the complete media fragment information of the currently received media information packet can be obtained, so that in the subsequent steps, the various media fragment information of the same time period can be aggregated into a complete free viewpoint fragment.

[0068] In one embodiment, the information fragmentation conditions can be set accordingly based on the specific scenario and are not limited here. For example, the information fragmentation conditions may include, but are not limited to: the ratio of the difference between the second display timestamp and the starting display timestamp to a preset time base is greater than or equal to the preset fragmentation duration, wherein the preset time base may be, but is not limited to, the time base of the corresponding media information stream. When the duration of all media information packets is the same, the duration of the media information packet may be, but is not limited to, set to the preset fragmentation duration. The difference between the two display timestamps is used to measure the degree of difference between the second display timestamp and the starting display timestamp. In other words, if the second display timestamp is large enough to further implement subsequent information fragmentation, then when the ratio of the difference between the second display timestamp and the starting display timestamp to the preset time base is less than the preset fragmentation duration, it can be determined that the currently received media information packet does not need to be fragmented.

[0069] In one embodiment, the second display timestamp of the currently received media information packet can be obtained in various ways, which are not limited herein. For example, the display timestamps of all media information packets can be aggregated and then compared to obtain the second display timestamp of the currently received media information packet.

[0070] In one embodiment, after subsequent information segmentation is performed according to the new start display timestamp, the next information segmentation can be continued in the manner of step S142. That is, when the duration of the subsequent information segment is clear, the next start display timestamp can be determined based on the duration of the information segment, the previous start display timestamp and the preset time reference, so that subsequent information segmentation can be performed based on the next start display timestamp.

[0071] like Figure 4As shown, an embodiment of the present application further illustrates the steps before steps S141 to S142, including but not limited to steps S160 to S180.

[0072] Step S160: Detecting whether a first target media information flow exists, wherein the first target media information flow is a media information flow that meets the interruption recovery condition;

[0073] Step S170: When the presence of a first target media information stream is detected, obtaining a difference between a second display timestamp corresponding to the first target media information stream and a start display timestamp corresponding to a plurality of second target media information streams, wherein the second target media information stream is a media information stream that does not meet the interruption recovery condition;

[0074] Step S180: Update the start display timestamp and fragment sequence number of the first target media information stream to the start display timestamp and fragment sequence number of the second target media information stream corresponding to the target difference value, wherein the target difference value is the smallest one among all the differences.

[0075] In this step, since interruption recovery affects subsequent information fragmentation of media information packets, in step S160, the interruption recovery status is further determined by detecting whether there is a first target media information stream that meets the interruption recovery condition. When the presence of the first target media information stream is detected, the difference between the second display timestamp corresponding to the first target media information stream and the start display timestamps corresponding to multiple second target media information streams is obtained. That is, considering the difference in display timestamps between the first target media information stream that meets the interruption recovery condition and all second target media information streams that do not meet the interruption recovery condition, the start display timestamp and fragment sequence number of the second target media information stream corresponding to the target difference are selected from all second target media information streams as the basis for updating the start display timestamp and fragment sequence number of the first target media information stream. Since the target difference is the smallest among all the differences, the start display timestamp and fragment sequence number of the first target media information stream can be updated to the start display timestamp and fragment sequence number of the nearest neighboring media information stream. This reduces the difficulty of subsequent information fragmentation, i.e., minimizes information fragmentation and reduces network bandwidth requirements.

[0076] In one embodiment, the interruption recovery condition can be set accordingly based on the specific scenario and is not limited here. For example, the interruption recovery condition may include, but is not limited to: the ratio of the difference between the second display timestamp and the display timestamp of the last received media information packet to a preset time reference is greater than a preset timeout period, wherein the preset time reference may be, but is not limited to, the time reference of the corresponding media information stream. By comparing the difference between the second display timestamp and the display timestamp of the last received media information packet, the difference between the second display timestamp of the currently received media information packet and the display timestamp of the most recently received media information packet can be measured to better determine the actual timeout degree of the second display timestamp. It can be understood that when the ratio of the difference between the second display timestamp and the display timestamp of the last received media information packet to the preset time reference is less than or equal to the preset timeout period, it can be determined that interruption recovery is not required for the currently received media information packet.

[0077] Step S150: Aggregate target media segment information in all media information streams to obtain free-viewpoint media segment information, wherein the target media segment information is media segment information having the same segment sequence number.

[0078] In this step, the first display timestamp of the acquired target media information packet is uniformly set as the starting display timestamp of each media information stream to solve the defect that the images of each media information stream arrive at the media server inconsistently at the same time. In this case, each media information stream is segmented according to the starting display timestamp to obtain multiple media segmentation information, and the media segmentation information with the same segmentation sequence number in all media information streams is aggregated to obtain complete free viewpoint media segmentation information, thereby ensuring the image quality while avoiding large-scale spatial jumps in the video image during the user's perspective switching process; therefore, the embodiment of the present application enables users to achieve seamless switching between free viewpoints, improves the user's video experience, and thus fills the technical gaps in related methods.

[0079] In one embodiment, the target media segment information may be, but is not limited to, the media segment information with a segment number other than 1. Figure 2a and Figure 2b As can be seen from the example, the display timestamp of the first media fragment information in the media information stream of each camera position is modified to the first display timestamp of the first received media information packet. In this case, the duration of the first media fragment information of each camera position (that is, the media fragment information with fragment number 1) is not the same. If the media fragment information with fragment number 1 is directly aggregated, it will not correspond. Therefore, aggregation can be started from the media fragment information with fragment number 2 to obtain reliable and stable free viewpoint media fragment information.

[0080] In one embodiment, there is no need to compress the video streams of each camera and then splice them into an ultra-high-resolution large picture and then perform image correction as in the prior art. Instead, the media information stream is segmented according to the corresponding display timestamp and aggregated based on the target media segment information in the media information stream to obtain the final free-viewpoint media segment information. This can greatly reduce the requirements for network bandwidth and is more suitable for users. Moreover, by using the media segment information splicing method of the embodiment of the present application, there is no need to consider the actual impact of the resolution of each camera, that is, there is no need to reduce the resolution of each camera by adapting to the resolution of user playback, thereby further improving the user experience.

[0081] like Figure 5 As shown, an embodiment of the present application further illustrates step S150, which includes but is not limited to steps S151 to S153.

[0082] Step S151: traverse the target media segment information in each media information stream in sequence;

[0083] Step S152: Determine whether the current target media segment information is the first media segment information after the interruption is restored;

[0084] Step S153: If the current target media segment information is not the first media segment information after the interruption is recovered, the current target media segment information is aggregated.

[0085] In this step, the target media segment information in each media information stream is traversed to determine whether the current target media segment information is the first media segment information after the interruption is recovered. According to the relevant comments of the above embodiment, since the first media segment information after the interruption is recovered is similar to the media segment information whose first display timestamp of each camera is modified, it is not well suitable for aggregation. Therefore, when it is determined that the current target media segment information is not the first media segment information after the interruption is recovered, the current target media segment information is selected for aggregation to obtain reliable and stable free viewpoint media segment information. That is to say, for the interruption recovery situation, it is not aggregated until at least the second media segment information after the interruption is recovered, so that the free viewpoint media segment information is obtained better.

[0086] In one embodiment of the present application, step S150 is further described based on steps S151 to S153 , and step S150 also includes but is not limited to step S154 .

[0087] Step S154: If the current target media segment information is the first media segment information after the interruption is recovered, the current target media segment information is not aggregated.

[0088] In this step, since the first media segment information after the interruption is recovered is similar to the media segment information whose first display timestamp of each camera position is modified, it is not suitable for aggregation. Therefore, when it is determined that the current target media segment information is the first media segment information after the interruption is recovered, the current target media segment information will not be aggregated to avoid affecting the overall aggregation process of the free viewpoint media segment information. That is to say, for the interruption recovery situation, it will not be aggregated until at least the second media segment information after the interruption is recovered, so that the free viewpoint media segment information is better obtained.

[0089] Several specific examples are given below to illustrate the working principles and processes of the above embodiments.

[0090] Example 2:

[0091] like Figure 6 As shown, Figure 6 A schematic diagram of a media server for executing a media information processing method provided in accordance with an embodiment of the present application.

[0092] Reference Figure 6 The media server may include, but is not limited to, a stream receiving module, an alignment module, and a splicing module, wherein:

[0093] The receiving module is used to pull media streams from each camera position at the front end of the free viewpoint (i.e. Figure 6 The media streams of camera 1, camera 2, camera 3, ... camera n are added to the receiving stream buffer queue;

[0094] The alignment module is used to take out the media stream from the receiving stream buffer queue, align it and then segment it;

[0095] The splicing module is used to aggregate various camera positions according to the same fragment sequence number into a complete free viewpoint fragment.

[0096] According to the above examples, through the cooperation of the stream receiving module, the alignment module and the splicing module, users can achieve seamless switching between free viewpoints, improve the user's video experience, and thus fill the technical gaps in related methods.

[0097] Example 3:

[0098] The following is a detailed description of the working principle and process of the alignment module in Example 2.

[0099] like Figure 7 As shown, Figure 7 A flowchart of a method for processing media information executed by an alignment module provided in one embodiment of the present application.

[0100] Reference Figure 7 , the alignment module may, but is not limited to, perform the following steps:

[0101] Step a: Traverse each media information packet in the receiving stream cache queue to determine whether the current media information packet is the first media information packet received. If so, force the startpts of the first fragment of all camera positions to be set to the PTS of the first media information packet received (i.e., the current media information packet), and then proceed to step b. Otherwise, no processing is performed and step b is directly entered.

[0102] Step b: Store the media information packets into the linked lists corresponding to the camera positions.

[0103] Step c: Determine whether there is a scenario where the camera position needs to recover from interruption based on the formula (curpts-lastpts) / timebase>overtime. If so, proceed to step d; otherwise, proceed to step e, where curpts represents the PTS of the current media information packet of the camera position, lastpts represents the PTS of the previous media packet of the camera position, timebase represents the time base of the media stream, and overtime represents the preset timeout period.

[0104] Step d: Calculate the difference diffpts between curpts and the startpts of other normal cameras, find the startpts and segno (segno refers to the fragment sequence number, starting from 1 and increasing) of the camera corresponding to the minimum diffpts, and set it as the corresponding information in the interruption recovery camera; specifically, refer to Figure 8 , Figure 8 A schematic diagram of multiple media information streams provided in another embodiment of the present application. The numbers in the boxes represent the PTS values ​​of the current media information packets. It can be seen that when the startpts of camera position 1 is 0, the segno is 1; when the startpts of camera position 2 is 540000, the segno is 2; when the startpts of camera position 3 is 108000, the segno is 3. If camera position 1 is interrupted, the PTS difference between it and the corresponding media information packets of camera positions 2 and 3 is calculated. The difference between it and the media information packets of camera position 2 is 1083600-5400. 00=543600, and the value between it and camera position 3 is 1083600-1080000=3600. That is to say, the camera position information startpts and segno corresponding to camera position 3 are set to the corresponding information in camera position 1. In this way, camera position 1 can be aligned with camera position 3 after the interruption is restored. When the next media information packet PTS=1080000 of camera position 2 arrives, camera position 2 switches to the next fragment, and startpts and segno are also aligned with camera position 3. This ensures that camera position 1 can be aligned with other camera positions after the interruption is restored.

[0105] Step e: Determine whether the camera position meets the conditions for segmentation based on the formula (curpts-startpts) / timebase>=min_seg_duration. If so, segment directly. The segment is named segno. After segno is increased by 1, go to step a. Otherwise, go directly to step a without any processing. Among them, min_seg_duration represents the preset segmentation duration.

[0106] Example 4:

[0107] The following is a detailed description of the working principle and process of the splicing module in Example 2.

[0108] like Figure 9 As shown, Figure 9 A flowchart of a method for processing media information executed by a splicing module provided in one embodiment of the present application.

[0109] Reference Figure 9 , the splicing module can, but is not limited to, perform the following steps:

[0110] Step a: Scan the fragment information to determine whether the fragment number n to be aggregated is 1. If not, proceed to step b. If so, add 1 to the fragment number and proceed to step a again. Since the length of the first fragment of each camera position is inconsistent after forced alignment, the first fragment of each camera position will not be aggregated.

[0111] Step b: traverse the slices with the same serial number of each camera position in turn, that is, traverse the slices with the serial number n of each camera position in turn, and determine whether the slice is the first slice after the interruption is restored. If so, go to step b again, otherwise go to step c.

[0112] Step c: Aggregate the slices of the camera position into free viewpoint media slice information, and determine whether all slices with camera position slice number n have been scanned. If so, add 1 to the slice number and go to step a; otherwise, go to step b.

[0113] In combination with the above examples, it can be seen that the embodiment of the present application forcibly sets the first PTS of the starting segment of all camera positions in the initialization phase, slices the segments according to the segment length and increments the segment sequence number in the operation phase, and when a scenario of interruption and recovery of the camera position is monitored, the first PTS and segment sequence number of the current segment of the camera position are recalculated, and then all camera position information of the same time period is aggregated into a complete free viewpoint segment according to the segment sequence number, so that the user can choose the perspective for playback. This can solve the problem of inconsistent images of each camera position's code stream arriving at the media server at the same time, avoid large-scale spatial jumps in the video image during the user's perspective switching process, while ensuring image quality, and reducing the bandwidth and performance requirements of the terminal device, so that the user can achieve seamless switching between free viewpoints and improve the user's video experience.

[0114] The method of the embodiment of the present application can be widely used in panoramic video generation in VR, virtual viewpoint scenes, etc.

[0115] In addition, if Figure 10 As shown, an embodiment of the present application also discloses a media information processing device 100, including: at least one processor 110; at least one memory 120, for storing at least one program; when the at least one program is executed by the at least one processor 110, a media information processing method as in any of the previous embodiments is implemented.

[0116] In addition, an embodiment of the present application further discloses a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are used to execute the media information processing method in any of the previous embodiments.

[0117] In addition, an embodiment of the present application also discloses a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. The processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device executes the media information processing method as in any of the previous embodiments.

[0118] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A method for processing media information, comprising: receiving a plurality of media information streams, wherein the media information streams include a plurality of media information packets; Obtaining a first display timestamp of a received target media information packet, wherein the target media information packet is a first received media information packet among all the media information packets; Using the first display timestamp as the start display timestamp of each of the media information streams; Segmenting each of the media information streams according to the start display timestamp to obtain a plurality of media segment information of each of the media information streams, wherein the media segment information corresponds to a segment sequence number, and all the media segment information with the same segment sequence number have the same media duration; The target media segment information in all the media information streams is aggregated to obtain free viewpoint media segment information, wherein the target media segment information is the media segment information having the same segment sequence number.

2. The media information processing method according to claim 1, wherein: The fragmenting of each of the media information streams according to the start display timestamp includes: For each of the media information streams, obtain the second display timestamp of the currently received media information packet. When it is determined that the information fragmentation condition is met based on the second display timestamp and the starting display timestamp, perform initial information fragmentation based on the currently received media information packet, use the second display timestamp as the new starting display timestamp, and perform subsequent information fragmentation based on the new starting display timestamp.

3. The media information processing method according to claim 2, wherein: The information fragmentation conditions include: A ratio of a difference between the second display timestamp and the start display timestamp to a preset time base is greater than or equal to a preset segment duration.

4. The media information processing method according to claim 2, wherein: Before segmenting each of the media information streams according to the start display timestamp, the media information processing method further includes: Detecting whether a first target media information flow exists, wherein the first target media information flow is the media information flow that meets the interruption recovery condition; When the presence of the first target media information stream is detected, obtaining a difference between the second display timestamp corresponding to the first target media information stream and the start display timestamps corresponding to a plurality of second target media information streams, wherein the second target media information stream is the media information stream that does not meet the interruption recovery condition; The start display timestamp and the fragment sequence number of the first target media information stream are updated to the start display timestamp and the fragment sequence number of the second target media information stream corresponding to the target difference value, wherein the target difference value is the smallest one among all the differences.

5. The media information processing method according to claim 4, characterized in that: The interruption recovery conditions include: A ratio of a difference between the second display timestamp and the display timestamp of the last received media information packet to a preset time base is greater than a preset timeout duration.

6. The media information processing method according to claim 1, wherein: The target media segment information is the media segment information whose segment sequence number is not 1.

7. The media information processing method according to claim 6, characterized in that: The aggregating target media segment information in all the media information streams includes: Traversing the target media segment information in each of the media information streams in sequence; Determining whether the current target media segment information is the first media segment information after the interruption is restored; If the current target media segment information is not the first media segment information after the interruption is recovered, the current target media segment information is aggregated.

8. The media information processing method according to claim 7, wherein: The aggregating target media segment information in all the media information streams further includes: If the current target media segment information is the first media segment information after the interruption is recovered, the current target media segment information is not aggregated.

9. A media information processing device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the media information processing method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the media information processing method according to any one of claims 1 to 8.

Images