Image frame processing method, network device, storage medium, and program product

Abstract

The present application provides an image frame processing method, a network device, a computer readable storage medium, and a computer program product. The method comprises: determining a redundant image frame on the basis of a target scenario where a current image frame is located and a frame type of the current image frame (S101); and discarding the current image frame and the redundant image frame (S102).

Description

Image frame processing method, network device, storage medium and program product

[0001] Cross-reference to related applications

[0002] The present application is based on and claims priority to Chinese Patent Application No. 202410744560.1, filed on June 7, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of communication technology, and in particular to an image frame processing method, a network device, a computer readable storage medium and a computer program product. BACKGROUND

[0004] In the encoding of a video service, the image frames of a video are often divided into a plurality of groups of pictures (GOPs), each GOP containing an intra-coded picture (I frame) and a plurality of predictive frames (P frames). The I frame is usually referred to as an I frame, which is usually the first frame of video encoding, and is characterized by containing a complete image information. In the decoding process, only the current frame needs to be decoded to extract a complete picture. If an I frame is lost in a video, the entire video will be in a black screen state, and the subsequent video cannot be normally played. The P frame is usually referred to as a P frame, and its characteristic is that it needs to refer to the previous encoded frame to correctly decode the image. When an abnormal situation occurs during video transmission, the image frame with transmission abnormality can be discarded according to the importance of the frame. At this time, if the associated frame of the current discarded image frame continues to be retained in the GOP, it cannot be decoded, which will cause redundancy and waste of air interface bandwidth. SUMMARY

[0005] Embodiments of the present application provide an image frame processing method, a network device, a computer readable storage medium and a computer program product.

[0006] In a first aspect, embodiments of the present application provide an image frame processing method, which includes the following steps: determining a redundant image frame according to a target scene and a frame type in which a current image frame is located; discarding the current image frame and the redundant image frame.

[0007] In a second aspect, embodiments of the present application provide a network device, which includes: one or more processors; a memory having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the image frame processing method as described in the first aspect.

[0008] In a third aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the image frame processing method according to the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. The computer program is executed by a processor to implement the image frame processing method according to the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0010] FIG. 1 is a schematic diagram of a structure of a GOP;

[0011] FIG. 2 is a schematic diagram of an application environment of an image frame processing method according to an embodiment of the present application;

[0012] FIG. 3 is a schematic diagram of steps of an image frame processing method according to an embodiment of the present application;

[0013] FIG. 4 is a schematic diagram of steps of a method for obtaining PDU Set QoS parameters according to an embodiment of the present application;

[0014] FIG. 5 is a schematic diagram of steps of a method for obtaining image frame information according to an embodiment of the present application;

[0015] FIG. 6 is a schematic diagram of a structure of a GTP-U extension header according to an embodiment of the present application;

[0016] FIG. 7 is a schematic diagram of steps of an image frame processing method according to an embodiment of the present application;

[0017] FIG. 8 is a schematic diagram of steps of an image frame processing method according to an embodiment of the present application;

[0018] FIG. 9a is a schematic diagram of steps of an image frame processing method according to an embodiment of the present application;

[0019] FIG. 9b is a schematic diagram of steps of an image frame processing method according to an embodiment of the present application;

[0020] FIG. 10a is a schematic diagram of steps of an image frame processing method according to an embodiment of the present application;

[0021] FIG. 10b is a schematic diagram of steps of an image frame processing method according to an embodiment of the present application;

[0022] FIG. 11 is a schematic diagram of a structure of an image frame processing system according to an embodiment of the present application;

[0023] FIG. 12 is a schematic diagram of steps of an XR packet processing flow according to an embodiment of the present application;

[0024] FIG. 13 is a structural schematic diagram of a network device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order for those skilled in the art to better understand the technical solutions of the present application, the technical solutions provided by the present application are described in detail below with reference to the drawings.

[0026] In the following, example embodiments will be described more fully with reference to the accompanying drawings, in which example embodiments can be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided as a full and enabling disclosure of the application, and to fully convey the scope of the application to the skilled in the art.

[0027] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0029] In the following description, reference is made to the "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets as each other and can be combined with each other without conflict.

[0030] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present application, and will not be interpreted in an overly literal or overly formal sense unless expressly so defined herein.

[0031] For the convenience of better understanding the solutions of the embodiments of the present application, first, the related art is introduced as follows.

[0032] In the encoding of video service, the image frames of the video are often divided into multiple groups of pictures (GOP), as shown in FIG. 1, each GOP contains an I frame and multiple P frames. The I frame is usually also called an internal picture, which is usually the first frame of video encoding, and its biggest feature is that it carries a complete image information, and in the decoding process, only the decoding of the frame is needed to completely extract a complete picture. Assuming that an I frame is lost in a video, the entire video will be in a black screen state, and the following video cannot be normally played out. The P frame is also called a forward reference frame, and the feature of this frame is that it needs to refer to the image information of the previous I frame to correctly decode the image. When an exception occurs in the video transmission process, such as transmission error or transmission congestion, the image frame can be discarded according to the importance of the frame. At this time, if the associated frame of the current discarded image frame continues to remain in the GOP and cannot be decoded, redundancy will be caused, wasting the air interface bandwidth.

[0033] The video service described in the embodiments of the present application can be an extended reality (Extended Reality, XR) service. In 3GPP technical specifications and 3GPP technical reports, some optimization improvements made by XR bearer in the 5G system are explained, which include frame integrity transmission function and congestion frame loss function, and the basis of the two functions is to define an image frame as a set of protocol data units (Protocol Data Unit Set, PDU Set). In an example, a series of IP packets generated after encoding a certain frame image belong to a certain PDU Set, that is, the IP packets of image I frame and P frame are a PDU Set, and the 3GPP standard defines a PDU Set integrity transmission indication for the XR service flow, which is carried to the base station through the protocol data unit set integrated handling indication (PDU Set Integrated Handling Indication, PSIHI) field in the PDU Set quality of service (Quality of Service, QoS) information when the XR service flow is established; the importance (Importance bits) attribute is defined for the PDU Set, which is carried to the base station through the general packet radio service tunnel protocol user plane (General Packet Radio Service Tunnel Protocol User Plane, GTP-U) extension header; the PSIHI is used to indicate whether the base station performs frame integrity transmission processing on the PDU Set when the network has IP packet transmission errors. The Importance bits are 4 bits, the value is 0 to 15, which is used to represent the relative importance between PDU Sets in the service flow, and the smaller the value is, the higher the importance is. Through the cooperation of PSIHI and importance, when there is a transmission error in an IP packet in the PDU Set or network transmission congestion occurs, the entire PDU Set is discarded according to the importance of the PDU Set, that is, the current frame image is discarded, but the protocol does not give specific information on how to discard the redundant frame according to the importance.

[0034] The above method can reduce the redundant transmission of IP packets in the air interface of the base station to some extent, and improve the use efficiency of the air interface wireless resources. However, the dependence between I frames and P frames is not fully utilized, and only the current frame is discarded. When the I frame transmission is incorrect, only the I frame is discarded, and the subsequent P frame cannot be discarded. Since the coding and decoding of the P frame in the subsequent GOP depend on the I frame, the undiscarded P frame cannot be decoded, thereby wasting the air interface bandwidth. When the P frame transmission is incorrect, the current frame is discarded according to the PSI HI indication. When the coding and decoding of the subsequent P frame have a dependence on the adjacent frame, that is, depend on the previous P frame that needs to be discarded, the undiscarded P frame cannot be decoded, thereby wasting the air interface bandwidth. Especially when the number of image frames in the GOP is large and the video client does not request the I frame retransmission, the air interface overhead is huge, and the image is seriously stuck. Similarly, when network congestion occurs and frames are lost, only the importance of the frame is considered for frame loss, and similar problems can occur.

[0035] Based on this, the present application provides an image frame processing method, a network device, a computer readable storage medium and a computer program product, which aims to reduce the air interface wireless resource overhead of the network device and improve the capacity of the cell video service.

[0036] Please refer to FIG. 2, which is an application environment schematic diagram of the image frame processing method provided by the present application. As shown in FIG. 2, the application environment of the image frame processing method provided by the present application includes but is not limited to: a terminal, a network device, a core network and an extended reality system.

[0037] The terminal is responsible for receiving data from the extended reality system or sending data to the extended reality system. In some embodiments, the terminal can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, a personal digital assistant (PDA) and the like.

[0038] The network device is responsible for receiving the service bearer establishment, deletion and other management functions sent by the core network, and is also responsible for the data sent by the terminal and the extended reality system, and performs uplink and downlink scheduling respectively. When scheduling, the frequency spectrum resource of the cell where the terminal is located, i.e., the resource block (RB) is used for scheduling. The RB resource is limited by the cell frequency spectrum bandwidth and is limited, and improving the use efficiency of the RB resource and avoiding waste of the RB resource are important functions of the network device. In some embodiments, the network device can be a new radio access technology (NR) base station (next generation Node B, gNB) or a small station, a micro station, and can also be a relay station, a transmission and reception point (TRP), a road side unit (RSU), etc.

[0039] The core network is responsible for terminal registration, mobility management, service bearer management, interaction with the extended reality system, and obtaining the quality of service (QoS) requirements of the XR service. The core network element described in the embodiments of the present application can be a 5G core network element in the 5G system, for example, it can be an access and mobility management (AMF) network element or a user plane function (UPF) network element, etc.

[0040] The extended reality system includes an extended reality application function responsible for interacting with the core network, negotiating the QoS requirements of the service flow and related parameters of the inter-frame dependency relationship, and an extended reality server responsible for the transmission and reception of XR service data, etc.

[0041] The air interface connection between the terminal and the network device, and the interface connection between the network device, the core network and the extended reality system.

[0042] Please refer to FIG. 3, which is a step schematic diagram of an image frame processing method provided by the embodiments of the present application. The method can be applied in the application environment shown in FIG. 2 and executed by the network device. As shown in FIG. 3, the image frame processing method provided by the embodiments of the present application includes but is not limited to the following steps S101 and S102.

[0043] Step S101, determining a redundant image frame according to a target scene and a frame type in which a current image frame is located.

[0044] It should be noted that in the embodiments of the present application, the image frames to be discarded are not the same in different target scenarios and different frame types. The target scenarios can include a frame transmission error scenario and a frame transmission congestion scenario, and the frame types can include an I frame and a P frame. The embodiments of the present application determine the redundant image frames to be discarded together with the current image frame according to the target scenario in which the current image frame is located and the frame type of the current image frame. The redundant image frames described in the embodiments of the present application represent image frames that are coded in dependence on the current image frame.

[0045] In step S102, the target image frame is discarded.

[0046] When it is confirmed that the current image frame meets the discard condition, the method of the embodiments of the present application can be used to determine whether there are redundant image frames according to the target scenario in which the current image frame is located and the frame type, and discard the current image frame and the redundant image frames together. In this way, redundant message transmission can be reduced to the greatest extent, air interface resources can be saved, and the use efficiency of air interface wireless resources can be improved.

[0047] The discard condition can include that the current image frame is in a target scenario, for example, the current image frame is in a frame transmission error scenario or a frame transmission congestion scenario. In combination with the type of the target scenario and the frame type of the current image frame, the image frame to be discarded can be determined.

[0048] In a possible implementation, the current image frame is in a frame transmission error scenario, and the current image frame is an I frame. Based on the scenario and the frame type, it is confirmed that the current I frame can be discarded, and all P frames between the current I frame and the next I frame are regarded as redundant frames, so that the redundant frames are discarded together with the current I frame.

[0049] In a possible implementation, the current image frame is in a frame transmission error scenario, and the current image frame is a P frame. Based on the scenario and the frame type, it is confirmed that the current P frame can be discarded. In this case, P frame interdependence information also needs to be obtained to determine whether subsequent P frames depend on the current P frame for decoding, that is, whether there is a forward dependency between P frames. If there is a forward dependency between P frames, all P frames between the current P frame and the next I frame are regarded as redundant frames, so that the redundant frames are discarded together with the current P frame.

[0050] In a possible implementation, the current image frame is in a frame transmission congestion scene, and the current image frame is a P frame. Based on the scene and the frame type, it is determined that the current P frame can be discarded. In this case, P inter-frame dependency information also needs to be acquired to determine whether a subsequent P frame depends on the current P frame for decoding, that is, whether there is a forward dependency between P frames. If there is a forward dependency between P frames, all P frames between the current P frame and a next I frame are regarded as redundant frames, and then the redundant frames are discarded together with the current P frame.

[0051] In an embodiment, the image frame processing method provided by the embodiment of the application further includes: acquiring a protocol data unit set quality of service (PDU Set Qos) parameter of the target service flow. In an implementation, the network device can acquire the PDU Set Qos parameter from a control plane network element of a core network through an N2 interface.

[0052] It can be understood that the current image frame and the redundant image frame described in the embodiment of the application are both located in the target service flow.

[0053] In an embodiment, the PDU Set Qos parameter includes P inter-frame dependency information. Based on the P inter-frame dependency information, it can be determined whether there is a forward dependency between adjacent P frames in a GOP, that is, whether a subsequent P frame depends on a previous P frame for decoding.

[0054] In an embodiment, the PDU Set Qos parameter includes protocol data unit set integrated processing indication (PSIHI) information, and the PSIHI information is used to indicate whether frame integrity transmission needs to be performed on the image frame. In a possible implementation, when the PSIHI information indicates that frame integrity transmission needs to be performed on the image frame, the step S101 and the step S102 shown in FIG. 3 are performed.

[0055] In an embodiment of the present application, the control plane network element of the core network obtains the PDU Set QoS parameter of the target service flow from the extended reality system, and the PDU Set QoS parameter includes PSIHI information and P interframe dependency information, etc. The PSIHI information is used to determine whether frame integrity transmission is required for image frames, and the P interframe dependency information is used to determine whether there is a forward dependency relationship between P frames. The PSIHI can include but is not limited to PDU Set Package Delay Budget (PSDB) and PDU Set Error Rate (PSER). After the control plane network element of the core network receives the PDU Set QoS parameter, it sends the received PDU Set QoS parameter to the network device, so that the network device can determine whether frame integrity transmission is required for image frames according to the PSIHI information in the PDU Set QoS parameter, and when it is determined that frame integrity transmission is required for image frames and the current P frame is in a target scenario that needs to be discarded, it determines whether there is a redundant image frame (i.e. redundant P frame) according to the P interframe dependency information in the PDU Set QoS parameter.

[0056] Please refer to FIG. 4, which is a step diagram for obtaining PDU Set QoS parameter provided by an embodiment of the present application. As shown in FIG. 4, the process of obtaining PDU Set QoS parameter can include but is not limited to steps S201 to S204.

[0057] Step S201, the extended reality system sends the PDU Set QoS parameter to the core network.

[0058] The PDU Set QoS parameter includes at least one of the PSIHI information and the P interframe dependency information.

[0059] In an embodiment, the extended reality system can be an extended reality application server or an extended reality application function network element. The core network receives the PDU Set QoS parameter sent by the extended reality system through the control plane network element (such as AMF).

[0060] Step S202, send the PDU Set QoS parameter response to the extended reality system.

[0061] In an embodiment, after the core network allocates the PDU Set QoS parameter, it replies to the extended reality system with a PDU Set QoS parameter response message to inform the extended reality system that the PDU Set QoS parameter has been received.

[0062] Step S203, the core network sends a control plane message establishment / modification request carrying the PDU Set QoS parameter to the network device.

[0063] In an embodiment, the control plane network element of the core network sends a control plane message establishment / modification request carrying the PDU Set QoS parameter to the network device (gNB) through a control plane interface (such as an N2 interface).

[0064] Step S204, the network device sends a control plane message establishment / modification request response to the core network after completing the establishment / modification.

[0065] In an embodiment, the network device (gNB) replies to the control plane message establishment / modification response to the control plane network element of the core network after receiving the PDU Set QoS parameter, to inform the core network that the PDU Set QoS parameter has been received.

[0066] After the network device receives the PDU Set QoS parameter from the control plane network element of the core network, it can extract at least one of the PSIHI information and the P inter-frame dependency information from the PDU Set QoS parameter.

[0067] It should be noted that the control plane message establishment / modification request sent by the core network to the network device in the embodiments of the present application can be a PDU session establishment / modification request, or other information, which is not limited in the present application.

[0068] FIG. 4 illustrates a way of obtaining PSIHI information and P inter-frame dependency information from a control plane. When the PDU Set QoS parameter is obtained, it can be determined whether the image frame needs frame integrity transmission processing and whether there is a forward dependency relationship between P frames when the image frame is transmitted.

[0069] In an embodiment, the image frame processing method provided by the embodiments of the present application comprises: receiving a target service flow, the target service flow comprising a plurality of image frames, each image frame being composed of a protocol data unit set (PDU Set).

[0070] The plurality of image frames in the target service flow can be divided into a plurality of picture groups (GOPs), each GOP containing an I frame and a plurality of P frames, each I frame / P frame corresponding to a PDU Set composed of a plurality of packets.

[0071] In an embodiment, the network device can receive the target service flow from the user plane network element (such as a UPF) of the core network through a user plane interface (such as an N3 interface), and identify and obtain the PDU Set corresponding to each image frame from the target service flow.

[0072] In an embodiment, the image frame processing method provided by the embodiment of the present application comprises: obtaining a PDU Set corresponding to a current image frame; and obtaining a frame type corresponding to the current image frame from a protocol packet header corresponding to the PDU Set.

[0073] The protocol packet header can be a general packet radio service tunneling protocol user plane extension header (GTP-U extension header). The frame type information corresponding to the image frame can be obtained from the GTP-U extension header corresponding to the image frame.

[0074] In an embodiment, when the frame type corresponding to the current image frame is a P frame, the image frame processing method provided by the embodiment of the present application comprises: obtaining a PDU Set corresponding to the current P frame; and obtaining P frame interdependence information from a protocol packet header corresponding to the PDU Set, wherein the P frame interdependence information is used to indicate whether there is a forward dependency between P frames.

[0075] In the embodiment of the present application, the P frame interdependence information can be obtained from a control plane of a core network or from a user plane of the core network. When obtained from the control plane of the core network, the P frame interdependence information can be obtained through a PDU Set QoS parameter. When obtained from the user plane of the core network, the P frame interdependence information can be obtained through a protocol packet header (for example, a GTP-U extension header) corresponding to the current P frame.

[0076] In a possible implementation, the image frame processing method provided by the embodiment of the present application comprises: receiving a target service stream sent by a user plane network element of a core network, wherein the target service stream comprises a plurality of image frames, and each image frame is composed of a PDU Set; obtaining a PDU Set corresponding to a current image frame after receiving the target service stream sent by the user plane network element of the core network; determining a frame type corresponding to the current image frame according to a GTP-U extension header corresponding to the PDU Set, so as to determine whether a discard condition is met according to a target scene in which the current image frame is located and the frame type.

[0077] In another possible implementation, the image frame processing method provided by the embodiment of the present application comprises: receiving a target service stream sent by a user plane network element of a core network, wherein the target service stream comprises a plurality of image frames, and each image frame is composed of a PDU Set; obtaining a PDU Set corresponding to a current image frame after receiving the target service stream sent by the user plane network element of the core network; determining a frame type corresponding to the current image frame according to a GTP-U extension header corresponding to the PDU Set, and when the frame type is a P frame, further obtaining P frame interdependence information from the GTP-U extension header, so as to determine an image frame that needs to be discarded according to a target scene in which the current P frame is located, the frame type, and the P frame interdependence information.

[0078] Please refer to FIG. 5, which is a schematic diagram of steps for acquiring image frame information according to an embodiment of the present application. As shown in FIG. 5, the process of acquiring image frame information can include, but is not limited to, steps S301-S303.

[0079] In step S301, a target service flow is received, and the target service flow includes a plurality of image frames, each of which is composed of a PDU Set.

[0080] In step S302, a PDU Set corresponding to a current image frame is acquired.

[0081] In step S303, image frame information is acquired from a protocol packet header corresponding to the PDU Set. The image frame information includes at least one of the following: a frame type corresponding to the current image frame, and inter-frame dependency information.

[0082] In an embodiment, the protocol packet header is a General Packet Radio Service Tunneling Protocol User Plane Extension Header (GTP-U Extension Header). The structure of the GTP-U Extension Header is shown in FIG. 6.

[0083] The second bit and the third bit of the first byte of the GTP-U Extension Header shown in FIG. 6 are reserved bits. In an embodiment of the present application, the reserved bit field in the first or second byte can be modified to a PDU Set Dependency Information field or the like, to transfer inter-frame dependency information, or can be modified to a frame type, with a value of 00 representing an I frame and a value of 01 representing a P frame, and other values being reserved.

[0084] In an embodiment, another aspect of the present application can also implicitly transfer a frame type through a frame importance field in the GTP-U Extension Header. For example, the PSI field in the GTP-U Extension Header maps a frame type to a certain importance value, with a value of 15 representing a P frame and a value of 0 representing an I frame. When a network device receives the importance value, it can determine whether the image frame is an I frame or a P frame.

[0085] In an embodiment, in another aspect, the payload of the GTP-U packet in the present application is a Real-Time Transport Protocol (RTP) packet. The type field of the Network Abstract Layer unit header (NAL unit header) in the RTP packet header can be used to determine the image frame type. Different type field values represent an I frame or a P frame. A network device can determine the I frame and P frame types by decoding the RTP header to acquire the type information.

[0086] The P frame received can also be determined whether it has a forward dependency relationship through the protocol packet header, so as to facilitate subsequent determination of the redundant image frame. For example, the P frame inter-dependence relationship information is carried through the reserved field of the GTP-U extension header.

[0087] Please refer to FIG. 7, which is a step schematic diagram of an image frame processing method provided by an embodiment of the present application. As shown in FIG. 7, the image frame processing method provided by the embodiment of the present application includes but is not limited to steps S401 to S403.

[0088] In step S401, the target scene where the current image frame is located and the frame type of the current image frame are acquired.

[0089] In step S402, in response to the target scene where the current image frame is located being a frame transmission error scene and the frame type of the current image frame being an I frame, all P frames between the current I frame and the next I frame are determined as redundant image frames.

[0090] The current I frame is the current image frame.

[0091] In step S403, the current I frame and the redundant image frames are discarded.

[0092] In the example of FIG. 7, if the current I frame is in the transmission error scene, the current I frame can be discarded. Considering that all P frames in the GOP depend on the current I frame for decoding, after the current I frame is discarded, all P frames in the GOP cannot be successfully decoded, so all P frames between the current I frame and the next I frame are determined as redundant image frames, which are discarded together with the current I frame, so as to release the air interface to the greatest extent and save resources.

[0093] Please refer to FIG. 8, which is a step schematic diagram of an image frame processing method provided by an embodiment of the present application. As shown in FIG. 8, the image frame processing method provided by the embodiment of the present application includes but is not limited to steps S501 to S502. In step S501, the target scene where the current image frame is located and the frame type of the current image frame are acquired.

[0094] In step S502, in response to the target scene where the current image frame is located being a frame transmission congestion scene and the frame type of the current image frame being an I frame, the current I frame is not discarded, but is kept waiting for continued scheduling.

[0095] In the example of FIG. 8, if the current image frame is in the frame transmission congestion scene and the current image frame is an I frame, the current I frame does not need to be discarded, but is kept waiting for continued scheduling and is transmitted when the air interface is idle.

[0096] Please refer to FIG. 9a, which is a schematic diagram of steps of an image frame processing method according to an embodiment of the present application. As shown in FIG. 9a, the image frame processing method according to the embodiment of the present application includes but is not limited to steps S601 to S604.

[0097] In step S601, the target scene in which the current image frame is located and the frame type of the current image frame are acquired.

[0098] In step S602, in response to the target scene in which the current image frame is located being a frame transmission error scene and the frame type of the current image frame being a P frame, P inter-frame dependency relationship information is acquired.

[0099] In step S603a, in response to the P inter-frame dependency relationship information indicating that there is a forward dependency between P frames, all P frames between the current P frame and the next I frame are determined as redundant image frames.

[0100] The current P frame is the current image frame.

[0101] In step S604, the current P frame and the redundant image frames are discarded.

[0102] It can be understood that the frame type and the P inter-frame dependency relationship information can be acquired by referring to the method described above, and thus will not be described herein again.

[0103] In the example of FIG. 9a, if the current P frame is in a transmission error scene, the current P frame can be discarded. Considering that there is a forward dependency between P frames in a GOP, after the current P frame is discarded, other P frames located after the current P frame in the GOP cannot be successfully decoded, and thus all P frames between the current P frame and the next I frame are determined as redundant image frames, which are discarded together with the current P frame, so as to release the air interface to the greatest extent and save resources.

[0104] Please refer to FIG. 9b, which is a schematic diagram of steps of an image frame processing method according to an embodiment of the present application. As shown in FIG. 9b, in a possible implementation manner, the image frame processing method according to the embodiment of the present application includes but is not limited to steps S601 to S603b.

[0105] In step S601, the target scene in which the current image frame is located and the frame type of the current image frame are acquired.

[0106] In step S602, in response to the target scene in which the current image frame is located being a frame transmission error scene and the frame type of the current image frame being a P frame, P inter-frame dependency relationship information is acquired.

[0107] In step S603b, in response to the P inter-frame dependency relationship information indicating that there is no forward dependency between P frames, only the current P frame is discarded.

[0108] The current P frame is the current image frame.

[0109] It can be understood that the frame type and the inter-P frame dependency information can be acquired by referring to the method described above, and will not be described herein again.

[0110] In the example of FIG. 9b, if the current P frame is in the transmission error scenario, the current P frame can be discarded. Since there is no forward dependency between the P frames in the GOP, that is, the other P frames after the current P frame in the GOP cannot be decoded successfully after the current P frame is discarded, all the P frames between the current P frame and the next I frame are determined as redundant image frames, and are discarded together with the current P frame, so as to release the air interface to the greatest extent and save resources.

[0111] Referring to FIG. 10a, a step schematic diagram of an image frame processing method provided by an embodiment of the present application is shown. As shown in FIG. 10a, the image frame processing method provided by the embodiment of the present application includes but is not limited to steps S701 to S704.

[0112] In step S701, the target scenario in which the current image frame is located and the frame type of the current image frame are acquired.

[0113] In step S702, in response to the target scenario in which the current image frame is located being a frame transmission congestion scenario and the frame type of the current image frame being a P frame, inter-P frame dependency information is acquired.

[0114] In step S703a, in response to the inter-P frame dependency information indicating that there is a forward dependency between the P frames, all the P frames between the current P frame and the next I frame are determined as redundant image frames.

[0115] In step S704, the current P frame and the redundant image frames are discarded.

[0116] In the example of FIG. 10a, if the current P frame is in the transmission error scenario, the current P frame can be discarded. Since there is no forward dependency between the P frames in the GOP, that is, the other P frames after the current P frame in the GOP cannot be decoded successfully after the current P frame is discarded, all the P frames between the current P frame and the next I frame are determined as redundant image frames, and are discarded together with the current P frame, so as to release the air interface to the greatest extent and save resources.

[0117] Referring to FIG. 10b, a step schematic diagram of an image frame processing method provided by an embodiment of the present application is shown. As shown in FIG. 10b, in a possible implementation manner, the image frame processing method provided by the embodiment of the present application includes but is not limited to steps S701 to S703b.

[0118] In step S701, the target scenario in which the current image frame is located and the frame type of the current image frame are acquired.

[0119] In step S702, in response to the target scenario in which the current image frame is located being a frame transmission congestion scenario and the frame type of the current image frame being a P frame, inter-P frame dependency information is acquired.

[0120] Step S703b, in response to the P inter-frame dependency relationship information indicating that the P inter-frames have no forward dependency, only the current P frame is discarded.

[0121] In the example of FIG. 10b, if the current P frame is in a transmission congestion scene, the current P frame can be discarded. Since there is no forward dependency among the P inter-frames in the GOP, that is, other P frames after the current P frame in the GOP do not depend on the current P frame for decoding, only the current P frame can be discarded.

[0122] Referring to FIG. 11, the embodiment of the present application further provides an image frame processing system, which comprises a scheduling processing module, a service identification module, a frame integrity transmission and congestion processing module, a session management module, a core network session management module, a data processing module and an extended reality system, wherein the scheduling processing module, the service identification module, the frame integrity transmission and congestion processing module and the session management module are located inside a network device. The functions of the modules are as follows.

[0123] The session management module is responsible for the session establishment of the XR service, receives the XR service parameters transmitted by the core network, and contains the inter-frame dependency relationship and related QoS parameters.

[0124] The core network session management module is located on the core network side, is responsible for the session establishment of the XR service, and cooperates with the session management module in the network device to establish the XR service. The XR service parameters obtained by interacting with the extended reality system are transmitted to the network device.

[0125] The data processing module is responsible for obtaining the XR service data from the extended reality system, obtaining some XR service parameters from the RTP packet of the extended reality system, and transmitting the XR service parameters to the network device through the GTP-U interface.

[0126] The service identification module is configured to identify the XR service, analyze the service characteristics, and transmit the service characteristics to the frame integrity transmission and congestion processing module. The service characteristics include the inter-frame dependency relationship, the frame type and frame size, the frame header, the frame tail and the like.

[0127] The frame integrity transmission and congestion processing module is configured to discard related frames according to the inter-frame dependency relationship when the XR service frames have transmission errors or the network congestion discards packets.

[0128] The scheduling processing module is configured to process the scheduling transmission and reception of the XR service, and report the PDU Set with transmission errors when there is a transmission error.

[0129] Based on the image frame processing module provided by the embodiment of the present application, the embodiment of the present application provides an image frame processing method, which refers to the following steps.

[0130] The session management module on the core network obtains the XR inter-frame dependency from the extended reality system, as well as other PDU Set QoS parameters such as PSIHI, including but not limited to PSDB (PDU Set Package Delay Budget), PSER (PDU Set Error Rate). These information and other QoS requirement information are passed to the session management module of the base station.

[0131] The session management module of the network device establishes an XR service flow session according to the QoS requirement information, and saves the inter-frame dependency.

[0132] The service identification module receives the inter-frame dependency, and obtains the frame type from the user plane message, determines the I-frame, P-frame properties, as well as the frame size, frame header, frame tail, frame sequence number and other information, and passes these information to the scheduling processing module and the frame integrity transmission and congestion processing module.

[0133] The scheduling processing module performs scheduling according to the frame type, frame header, frame tail, frame sequence number and other information. If there is an error in transmission or air interface congestion, and some XR service frames exceed the budget time of the XR image frame, it is transferred to the frame integrity transmission and congestion processing module for processing.

[0134] The frame integrity transmission and congestion processing module discards related frames according to the inter-frame dependency and frame type in the case of congestion and frame transmission error.

[0135] It should be noted that, please refer to FIG. 12, which is a step diagram of the XR message processing flow in the embodiment of the present application. The single message structure waiting for scheduling in the PDCP layer is shown in FIG. 12, and there are multiple messages in the queue. The continuous N messages belong to a PDU Set. The XR message processing flow is as follows.

[0136] Step S801, remove the GTP-U header from the GTP-U message, keep the PDU Set extension header, and pass it to the PDCP layer.

[0137] The PDU Set extension header contains the PDU Set sequence number PSSN.

[0138] Step S802, the PDCP layer adds a PDCP layer header in front of the message, which contains the PDCP sequence number SN1.

[0139] Step S803, PDCP message, remove the PDU Set extension header information, add RLC layer header in front of it, and send it to the MAC layer for processing. The RLC layer contains the sequence number SN of the RLC layer.

[0140] Step S804, the MAC layer adds a MAC header in front of the message, which saves the data source from which logical channel (PDCP queue).

[0141] Step S805, the MAC layer message is bound to a certain Harq channel for transmission, and the Harq channel is identified by a Harq ID.

[0142] It should be noted that when the message is transmitted on the Harq channel, if the message fails after being retransmitted on the Harq channel for multiple times, it is considered that the data transmission on the harq channel fails, and the above-mentioned message reverse index is used to confirm which PDU set appears transmission failure, and it can be considered that the image frame corresponding to the PDU set is in a frame transmission error scenario.

[0143] When the first message in the PDU set in the PDCP queue reaches the network device, a timer t1 is started for the PDU set, and the duration of t1 is PSDB. When the t1 timer expires and the PDU in the PDU set is not scheduled, that is, the end of PDU (EPDU) indication in the PDU Set extension header of the PDCP message associated with the PDU to be scheduled is 0, and the PDU Set ID is the same as the PDU set ID when the timer is started, the PDU Set is congested and expires, and it can be considered that the image frame corresponding to the PDU Set is in a frame transmission congestion scenario.

[0144] When the last PDU in the PDU Set is scheduled, that is, the end of PDU (EPDU) indication in the PDU Set extension header is 1, the timer is released, and it is considered that the PDU Set is normally scheduled successfully.

[0145] The embodiment of the present application also provides a network device, as shown in Figure 13, the network device 1400 comprises:

[0146] one or more processors 1410;

[0147] a memory 1420, one or more programs are stored on the memory 1420, when the one or more programs are executed by the one or more processors 1410, the one or more processors 1410 implement the image frame processing method provided by any embodiment of the present application.

[0148] The memory 1420 is a kind of non-transient network system, and can be used to store non-transient software programs and non-transient computer executable programs.In addition, the memory 1420 can include high-speed random access memory, and can also include non-transient memory, such as at least one magnetic disk storage device, flash memory device or other non-transient solid-state memory device.In some embodiments, the memory 1420 includes memory 1420 remotely arranged relative to the processor 1410, and these remote memories 1420 can be connected to the processor 1410 through a network.The above-mentioned network includes but is not limited to Internet, enterprise intranet, local area network, mobile communication network and combination thereof.

[0149] The memory 1420 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1420 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 1420 and are invoked and executed by the processor 1410 to perform the method of the embodiments of the present application.

[0150] The processor 1410 can be implemented in the form of a general-purpose CPU (central processing unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0151] In some embodiments, the network device further comprises:

[0152] The input / output interface is configured to implement information input and output;

[0153] The communication interface is configured to implement the communication interaction between the device and other devices, which can be implemented by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0154] The bus transmits information between various components (such as the processor 1410, the memory 1420, the input / output interface, and the communication interface) of the device.

[0155] The processor 1410, the memory 1420, the input / output interface, and the communication interface can be connected to each other for internal communication in the device through the bus.

[0156] An embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are used to execute the image frame processing method provided by any of the embodiments of the present application.

[0157] An embodiment of the present application further provides a computer program product, which comprises a computer program or computer instructions, and the computer program or computer instructions are stored in a computer readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device executes the image frame processing method provided by any of the embodiments of the present application.

[0158] In the embodiments of the present application, the redundant image frame can be determined according to the target scene where the current image frame is located and the frame type of the current image frame, and the current image frame and the determined redundant image frame are discarded together, so that the image frames associated with the current image frame can be prevented from being retained in the GOP after the current image frame is discarded, thereby avoiding the problem of redundancy. The embodiments of the present application can minimize the transmission of redundant messages, save air interface resources, and improve the use efficiency of air interface wireless resources.

[0159] The system architecture and application scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It can be understood by those skilled in the art that, as the system architecture evolves and new application scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0160] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In the embodiments provided by the present application, any reference to the memory, storage, database or other medium can include non-volatile and / or volatile memory. The non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0161] As will be appreciated by one of ordinary skill in the art, all or some of the steps, systems, and techniques disclosed herein can be embodied in software, firmware, hardware, and / or suitable combination thereof. Some or all of the physical components can be implemented in software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or in hardware, or in an integrated circuit, such as an application- specific integrated circuit. Such software can be distributed on computer readable media, which can comprise computer storage media (or non-transitory media), and communication media (or transitory media). As will be appreciated by one of ordinary skill in the art, the term computer storage media includes all physical and tangible computer storage media, such as a volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, as will be appreciated by one skilled in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0162] The above description is not intended to limit the scope of the application. Modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and spirit of the application shall fall within the scope of the application.

Claims

1. A method for processing image frames, comprising the steps of: determining a redundant image frame according to a target scene and a frame type in which a current image frame is located; and discarding the current image frame and the redundant image frame.

2. The image frame processing method of claim 1, wherein, The current image frame is located in a group of images, and the group of images comprises I frames and P frames, and the determining of the redundant image frame according to the target scene and the frame type in which the current image frame is located comprises: in response to the target scene in which the current image frame is located being a frame transmission error scene and the frame type of the current image frame being an I frame, determining all P frames between the current I frame and a next I frame as the redundant image frame.

3. The image frame processing method of claim 1, wherein, The current image frame is located in a group of images, and the group of images comprises I frames and P frames, and the determining of the redundant image frame according to the target scene and the frame type in which the current image frame is located comprises: in response to the target scene in which the current image frame is located being a frame transmission error scene and the frame type of the current image frame being a P frame, determining P inter-frame dependency information; in response to the P inter-frame dependency information indicating that there is a forward dependency between P frames, determining all P frames between the current P frame and a next I frame as the redundant image frame.

4. The image frame processing method of claim 1, wherein, The current image frame is located in a group of images, and the group of images comprises I frames and P frames, and the determining of the redundant image frame according to the target scene and the frame type in which the current image frame is located comprises: in response to the target scene in which the current image frame is located being a frame transmission congestion scene and the frame type of the current image frame being a P frame, determining P inter-frame dependency information; in response to the P inter-frame dependency information indicating that there is a forward dependency between P frames, determining all P frames between the current P frame and a next I frame as the redundant image frame.

5. The image frame processing method of claim 1, wherein, The current image frame is located in a group of images, and the group of images comprises I frames and P frames, and the method further comprises: in response to the target scene in which the current image frame is located being a frame transmission error scene and the frame type of the current image frame being a P frame, determining P inter-frame dependency information; in response to the P inter-frame dependency information indicating that there is no forward dependency between P frames, discarding only the current P frame.

6. The image frame processing method of claim 1, wherein, The current image frame is located in a group of images, and the group of images comprises I frames and P frames, and the method further comprises: in response to the target scene in which the current image frame is located being a frame transmission congestion scene and the frame type of the current image frame being a P frame, determining P inter-frame dependency information; in response to the P inter-frame dependency information indicating that there is no forward dependency between P frames, discarding only the current P frame.

7. The image frame processing method of claim 1, wherein, Before the determining of the redundant image frame according to the target scene and the frame type in which the current image frame is located, the method further comprises: obtaining a protocol data unit set quality of service (PDU Set Qos) parameter of a target service flow, wherein the current image frame and the redundant image frame are both located in the target service flow.

8. The image frame processing method of claim 7, wherein, The PDU Set Qos parameter comprises P inter-frame dependency information.

9. The image frame processing method of claim 7, wherein, The PDU Set Qos parameter further comprises protocol data unit set integrated processing indication (PSIHI) information, and the determining of the redundant image frame according to the target scene and the frame type in which the current image frame is located comprises: In response to the PSIHI information indicating that the image frame needs to be transmitted in frame integrity, determining a redundant image frame according to a target scene and a frame type in which the current image frame is located.

10. The image frame processing method of claim 1, wherein, Before determining the redundant image frame according to the target scene and the frame type in which the current image frame is located, the method further comprises: receiving a target service flow, the target service flow comprising a plurality of image frames, each of the image frames being composed of a PDU Set.

11. The image frame processing method of claim 10, wherein, After receiving the target service flow, before determining the redundant image frame according to the target scene and the frame type in which the current image frame is located, the method further comprises: obtaining a PDU Set corresponding to the current image frame; obtaining a frame type corresponding to the current image frame from a protocol packet header corresponding to the PDU Set.

12. The image frame processing method of claim 11, wherein, In a case where the frame type corresponding to the current image frame is a P frame, the method further comprises: obtaining P inter-frame dependency information from the protocol packet header corresponding to the PDU Set. 13.A network device, comprising: one or more processors; a memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image frame processing method according to any one of claims 1-12.

14. A computer readable storage medium storing a computer program, wherein, The computer program is executed by the processor to implement the image frame processing method according to any one of claims 1-12. 15.A computer program product, comprising a computer program, wherein when the computer program is executed by the processor, the image frame processing method according to any one of claims 1-12 is implemented.

Images