Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: establishing a target streaming media connection session for supporting transmission of streaming media data and image data based on a real-time streaming protocol; obtaining first media stream data sent by the server through the target streaming media connection session, and performing data display in a first display area based on the first media stream data, wherein the first media stream data is formed by a part of streaming media data requested in the target streaming media data according to a target playing command of the client; and obtaining second media stream data sent by the server through the target streaming media connection session, and performing data display in a second display area based on the second media stream data, wherein the second media stream data is formed by key frame image data of a specified preview time requested in the target streaming media data according to a target preview command of the client. According to the scheme, video playback and image preview can be carried out under the condition that the performance cost is not additionally increased.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a data processing method, apparatus, electronic device, and storage medium. Background Art

[0002] With the continuous development of video detection technology, there is often a need to play back multimedia data in the field of video detection. For example, a client can play back video data collected by an IPC (Internet Protocol Camera) and stored in an SD card by pulling the video data stream through streaming media technology and playing it back on the client. However, if you want to view preview images at any time, due to the device-side specification limitations, it is impossible to pull another data stream to achieve image preview. Although there are other devices that support multi-channel playback without specification limitations and can achieve image preview at any time by pulling another data stream, the video playback service ability will be affected, greatly reducing the video playback performance of the device. Summary of the Invention

[0003] The present invention provides a data processing method, apparatus, electronic device, and storage medium to achieve video playback and image preview simultaneously without additional performance costs.

[0004] In a first aspect, an embodiment of the present invention provides a data processing method applied to a client. The method includes:

[0005] Establish a target streaming media connection session between the client and the server based on the Real-Time Streaming Protocol (RTSP) to support the transmission of streaming media data and image data;

[0006] Obtain first media stream data sent by the server through the target streaming media connection session, and display the data in a first display area based on the first media stream data. The first media stream data is formed by a part of the streaming media data requested by the server in the target streaming media data based on the target play command of the client;

[0007] When obtaining the first media stream data sent by the server, obtain second media stream data sent by the server through the target streaming media connection session, and display the data in a second display area based on the second media stream data. The second media stream data is formed by key frame image data of a specified preview time requested by the server in the target streaming media data based on the target preview command of the client.

[0008] In a second aspect, an embodiment of the present invention further provides a data processing apparatus configured in a client. The apparatus includes:

[0009] A connection module, configured to establish a target streaming media connection session for supporting the transmission of streaming media data and image data between a client and a server based on the Real-Time Streaming Protocol;

[0010] A first display module, configured to obtain first media stream data sent by the server through the target streaming media connection session, and perform data display in a first display area based on the first media stream data, where the first media stream data is formed by a part of the streaming media data requested by the server from the target streaming media data based on a target play command of the client;

[0011] A second display module, configured to, when obtaining the first media stream data sent by the server, obtain second media stream data sent by the server through the target streaming media connection session, and perform data display in a second display area based on the second media stream data, where the second media stream data is formed by key frame image data of a specified preview time requested by the server from the target streaming media data based on a target preview command of the client.

[0012] In a third aspect, an electronic device is further provided in an embodiment of the present invention. The electronic device includes:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the data processing method described in any one of the above embodiments.

[0016] In a fourth aspect, a computer-readable medium is further provided in an embodiment of the present invention. The computer-readable medium stores computer instructions, and when the computer instructions are executed by a processor, the data processing method described in any one of the above embodiments is implemented.

[0017] In an embodiment of the present invention, a target streaming media connection session for supporting the transmission of streaming media data and image data is established between a client and a server based on the Real-Time Streaming Protocol (RTSP). The first media stream data sent by the server through the target streaming media connection session is obtained. Based on the target play command corresponding to the first media stream data, the requested partial streaming media data in the target streaming media data is displayed in a first display area. At the same time, the second media stream data is obtained from the server using the target streaming media connection session, and the key frame image data at the specified preview time requested in the target streaming media data based on the target preview command corresponding to the second media stream data is displayed in a second display area. In this solution, the RTSP protocol is extended and configured to support new media types, and the OPTIONS request method is added to the RTSP protocol. The key frame image data at the specified preview time requested by the preview command can be previewed after being transmitted through the target streaming media connection session used by the first media stream data. Without affecting the established media stream, the client can perform picture preview. Moreover, since the existing target streaming media connection session is used, only the performance overhead of sending image data is consumed to achieve preview of images at the specified preview time during video playback, reducing the unnecessary LAPI message interaction overhead or unnecessary cache overhead on the device side brought about by adding the image preview function.

[0018] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Brief Description of the Drawings

[0019] Combined with the accompanying drawings and referring to the following specific embodiments, the above and other features, advantages, and aspects of the embodiments of the present invention will become more obvious. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic, and the original components and elements are not necessarily drawn to scale.

[0020] Figure 1 is a flowchart of a data processing method provided by an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of the interface display of a client provided by an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of a protocol for media stream interaction provided by an embodiment of the present invention;

[0023] Figure 4a is a flowchart of the interaction of media stream data provided by an embodiment of the present invention;

[0024] Figure 4b is an interaction timing diagram of media stream data provided by an embodiment of the present invention;

[0025] Figure 5 is a schematic structural diagram of a data processing device provided by an embodiment of the present invention;

[0026] Figure 6 is a schematic structural diagram of an electronic device for implementing a data processing method provided by an embodiment of the present invention. Detailed implementation manners

[0027] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not used to limit the protection scope of the present invention.

[0028] It should be understood that the various steps recorded in the method embodiments of the present invention can be executed in different orders and / or executed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this regard.

[0029] The term "including" and its variants used herein are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0030] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions executed by these devices, modules or units or their interdependent relationships.

[0031] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] It is understandable that before using the technical solutions disclosed in the embodiments of the present invention, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present invention should be informed to users in an appropriate manner and the authorization of users should be obtained in accordance with relevant laws and regulations.

[0034] For example, when responding to an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that performs the operations of the technical solutions of the present invention according to the prompt message.

[0035] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving an active request from the user may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0036] Figure 1 The flowchart of a data processing method provided by an embodiment of the present invention is applicable to the situation of video playback and image preview on a client side. This method can be executed by a data processing device, which can be implemented in the form of software and / or hardware and is generally integrated in any electronic device with network communication functions. The electronic device can be a mobile terminal, a PC terminal, a server, etc.

[0037] As Figure 1 shown, the data processing method of the embodiment of the present invention may include the following processes:

[0038] S110. Establish a target streaming media connection session between the client and the server based on the Real-Time Streaming Protocol (RTSP) to support the transmission of streaming media data and image data.

[0039] Referring to Figure 2 , the client can play back the video data collected by the Internet Protocol Camera (IPC) and stored in the SD card, pull the video data through streaming media technology and play back the video data in the first display area 210 of the client. If you want to view the preview image at any time, you need to pull the second stream to obtain the preview picture at any time and display it in the second display area 211 corresponding to the timeline for preview. Although the second stream can be pulled to achieve the preview thumbnail image at any time, it will cause the video playback service ability to be halved, greatly reducing the playback performance of the device.

[0040] To this end, the Real-Time Streaming Protocol (RTSP) can be extended and configured to support new media types, so that a target streaming media connection session for supporting the transmission of streaming media data and image data can be established between the client and the server based on the RTSP. The server is a device that provides the video data collected by the network camera IPC, such as front-end devices like the network camera IPC device and the network video recorder NVR; the client is a device that plays back and displays the video data collected by the network camera IPC. Among them, the RTSP is used to enable users to control when playing real-time data downloaded from the Internet and is responsible for the requests and responses between the server and the client (including pausing, playing, fast-forwarding, rewinding, etc.).

[0041] As an optional but non-limiting implementation, establishing a target streaming media connection session for supporting the transmission of streaming media data and image data between the client and the server includes the following steps A1 - A3:

[0042] Step A1: Send a streaming media description command to the server so that the server sends target streaming media session description information to the client. The media type in the target streaming media session description information is a media type that supports the transmission of streaming media data and image data, and the target streaming media description information indicates the target streaming media data that needs to be transmitted in the server.

[0043] See Figure 3 、 Figure 4a and Figure 4b , the client connects to the server and sends a streaming media description command (DESCRIBE) based on the RTSP. The information received by the server mainly includes descriptions such as the data identifier (e.g., name), decoding type, and video resolution of the streaming media data. Then, a streaming media session description information (SDP) can be obtained from the server. The streaming media session description information indicates information such as the number of streams and the media type. The media type in the target streaming media session description information supports the transmission of streaming media data and image data.

[0044] Exemplarily, the client sends a streaming media description command (DESCRIBE) to the server, (DESCRIBE rtsp: / / 192.168.2.163 / media / video1 RTSP / 1.0), to obtain the streaming media session description information (SDP information) from the server. The media type of videoimage (image data intercepted from the streaming media data) is newly added to the SDP information of the server, so that the media types supported by the formed streaming media connection session include streaming media data and picture data. The media type enumeration value is 118 (the enumeration values adopted by different media types can be customized according to the media type requirements, as long as it is ensured that the media type can be represented by a unique matching enumeration value). The example is as follows:

[0045] DESCRIBErtsp: / / 192.168.2.163 / media / video1 / videoimage RTSP / 1.0

[0046] m=videoimage 0RTP / AVP 118

[0047] c=IN IP4 192.168.2.163

[0048] a=rtpmap:118H264 / 90000-IMAGE

[0049] a=control:rtsp: / / 192.168.2.163 / media / video1 / videoimage

[0050] a=recvonly

[0051] Step A2: According to the received target streaming media session description information, send a streaming media connection establishment command to the server to establish a target streaming media connection session between the client and the server for transmitting the target streaming media data of the server.

[0052] Step A3: After receiving the streaming media connection response message sent by the server, establish the target streaming media connection session between the client and the server.

[0053] See Figure 3 、 Figure 4a And Figure 4bThe client analyzes the streaming media session description information (SDP information) sent by the server, and sends a streaming media connection establishment command (SETUP) for each stream in the session. The streaming media connection establishment command tells the server the port used by the client to receive streaming media data. Exemplarily, the client continues to send the streaming media connection establishment command SETUP to the server to establish a streaming media connection session for sending the media type of videoimage to send streaming media data and picture data, and configures the session number of the streaming media connection session (session: 220540936). Among them, an example of the streaming media connection establishment command SETUP is: SETUP rtsp: / / 192.168.2.163 / media / video1 / videoimage RTSP / 1.0. After receiving the streaming media connection response message sent by the server, a target streaming media connection session is established between the client and the server.

[0054] S120. Obtain the first media stream data sent by the server through the target streaming media connection session, and perform data display on the first display area based on the first media stream data. The first media stream data is formed by the server based on the target play command of the client in the target streaming media data for the requested part of the streaming media data.

[0055] See Figure 3 、 Figure 4a And Figure 4b The client sends a play command (PLAY) to the server. The server starts to transmit the first media stream data to the client based on the target play command of the client in the requested part of the streaming media data in the target streaming media data. During the playback process, the client can also send commands to the server to control fast forward, rewind, pause, etc. Among them, the streaming media data includes continuously media data such as audio stream, video stream, text stream, image stream, animation stream, etc. in time.

[0056] As an optional but non-limiting implementation manner, obtaining the first media stream data sent by the server through the target streaming media connection session includes the following steps B1 - B2:

[0057] Step B1. Send a target play command to the server through the target streaming media connection session to instruct the server to encode and encapsulate the requested part of the streaming media data in the target streaming media data based on the target play command according to the Real-Time Transport Protocol; the play command is the command in the Real-Time Streaming Protocol originally used to instruct the server to start sending streaming media data to the client according to the transmission mechanism corresponding to the streaming media connection session.

[0058] Step B2. Receive the first media stream data sent by the server through the target streaming media connection session.

[0059] See Figure 3 、 Figure 4a and Figure 4b , the client continues to send a target play command PLAY command based on the Real-Time Transport Protocol (RTP) protocol to the server. The server will encode and encapsulate the requested part of the streaming media data in the target streaming media data based on the target play command of the client to form an RTP data packet. Then, the server starts sending the first media stream data to the client through the target streaming media connection session based on the formed RTP data packet. Exemplarily, after the server encodes and encapsulates the requested part of the streaming media data in the target streaming media data based on the target play command to form an RTP data packet, it sends the first media stream data including the RTP data packet to the client through the target streaming media connection session corresponding to the existing session number (session: 220540936).

[0060] See Figure 2 、 Figure 3 and Figure 4a , after receiving the first media stream data sent by the server, the client will decode the first media stream data and render it in the first display area of the client, so as to realize the playback display of the streaming media data corresponding to the target play command in the target streaming media data. Optionally, the client can send a termination command (TERADOWN) to the server to end the streaming media connection session.

[0061] As an optional but non-limiting implementation, based on the first media stream data for data display in the first display area, it includes the following steps:

[0062] Decode the first media stream data, and render and display it in the first display area based on the decoded first media data, so as to play back the streaming media data corresponding to the first media stream data.

[0063] S130. When obtaining the first media stream data sent by the server, obtain the second media stream data sent by the server through the target streaming media connection session, and perform data display in the second display area based on the second media stream data. The second media stream data is formed according to the key frame image data of the specified preview time requested by the server in the target streaming media data based on the target preview command of the client.

[0064] See Figure 2 、 Figure 3 and Figure 4a, add an options request command for obtaining image data of a specified preview time to the Real-Time Streaming Protocol (RTSP), denoted as the GET_IMGE command. The client continues to send the target preview command GET_IMGE based on the Real-Time Transport Protocol (RTP) to the server. When the client requests a preview of the key frame image data of the specified preview time in the target streaming media data through the target preview command, the server encodes and encapsulates the key frame image data requested in the target streaming media data based on the target preview command of the client to form an RTP data packet. Then, the server starts sending the second media stream data to the client through the target streaming media connection session based on the formed RTP data packet. Among them, the first media stream data and the second media stream data are different media stream data processed on the same media stream transmitted through the same target streaming media connection session.

[0065] As an optional but non-limiting implementation, obtaining the second media stream data sent by the server through the target streaming media connection session includes the following steps C1 - C2:

[0066] Step C1: Send a target preview command to the server through the target streaming media connection session to instruct the server to encode and encapsulate the key frame image data of the specified preview time requested in the target preview command in the target streaming media data based on the Real-Time Transport Protocol to obtain the second media stream data; the preview command is a newly added command in the Real-Time Streaming Protocol used to instruct the server to start sending the image data of the specified preview time to the client through the transmission mechanism corresponding to the streaming media connection session.

[0067] Step C2: Receive the second media stream data sent by the server through the target streaming media connection session.

[0068] See Figure 2 、 Figure 3 、 Figure 4a And Figure 4b , the client continues to send the target preview command GET_IMGE based on the Real-Time Transport Protocol (RTP) to the server. The preview command GET_IMGE contains the specified preview time previewtime: yyyy:mm:dd hh:mm:ss; an example of the preview command GET_IMGE is as follows:

[0069] GET_IMGErtsp: / / 192.168.2.163 / media / video1 / videoimage RTSP / 1.0

[0070] preview time="yyyy:mm:dd hh:mm:ss"

[0071] See Figure 2, Figure 3 , Figure 4a and Figure 4b After the server receives the target preview command GET_IMGE sent by the client, in the target streaming media data stored locally by the server, it retrieves the I-frame image data corresponding to the specified preview time (preview time), and encapsulates the corresponding I-frame image data into an RTP data packet, with the encoding format being JEPG / MJEPG (or a newly added encoding format). Furthermore, based on the formed RTP data packet, the server starts sending the second media stream data to the client through the target streaming media connection session.

[0072] As an optional but non-limiting implementation manner, performing data display on the second display area based on the second media stream data includes the following steps:

[0073] Decode the second media stream data, and perform rendering display on the second display area based on the key frame image data of the specified preview time after decoding, so as to preview the key frame image data corresponding to the second media stream data.

[0074] As an optional but non-limiting implementation manner, sending a target preview command to the server through the target streaming media connection session includes the following steps D1 - D2:

[0075] Step D1: In response to a trigger operation on the time axis corresponding to the target streaming media data, determine the specified preview time corresponding to the trigger operation.

[0076] Step D2: Send, through the target streaming media connection session, a target preview command for previewing and displaying the key frame image of the specified preview time in the target streaming media data to the server.

[0077] See Figure 2 There is a time axis corresponding to the target streaming media data displayed on the client. Through the trigger operation of moving the mouse to any point on the time axis, determine the specified preview time corresponding to the trigger operation. The client sends the newly added preview command to the server. The server retrieves the I-frame image data of the specified preview time point in the target streaming media data stored locally, encapsulates it into an RTP data packet, and then sends the second media stream data containing the RTP data packet to the client, realizing the separate parsing of the streaming media data and the key frame image data, decoding and displaying the streaming media data on the first display area for playback, and decoding and displaying the key frame image data on the second display area at the corresponding trigger position point on the time axis for preview display.

[0078] Optionally, the first display area corresponding to the client is the display window area for playing back and displaying the streaming media data collected by the network camera IPC and stored in the SD card. The second display area is a preview display window that pops up after triggering and selecting a specified preview time on the time axis corresponding to the target streaming media data and is used to render and display the image data of the specified preview time. Optionally, the position of the second display area is dynamically adjusted as the trigger operation position on the time axis corresponding to the target streaming media data. For example, the position of the second display area is adjusted as the selection position of the mouse on the time axis and is always directly above the mouse.

[0079] As an optional but non-limiting implementation manner, the data processing method of the embodiments of the present invention may include the following steps:

[0080] Detect the type of the media stream sent by the server through the target streaming media connection session. The type of the media stream can be distinguished based on the encoding and encapsulation format used by the media stream.

[0081] See Figure 2 、 Figure 3 、 Figure 4a And Figure 4b , if the type of the media stream parsed by the client through the target streaming media connection session is JEPG / MJEPG (or a newly added encoding format), it is parsed as the key frame image data for previewing the specified preview time on the time axis corresponding to the target streaming media data, and is rendered to the second display area and displayed to the user to implement image preview. If the media stream type sent by the client through the target streaming media connection session is H264 / H265, it is parsed as the streaming media data for video playback in the first display area of the client, continues to be rendered and displayed on the second display area, and is displayed to the user.

[0082] The technical solution of the embodiments of the present invention extends the configuration through the Real-Time Streaming Protocol (RTSP) protocol to support newly added media types, and adds an OPTIONS request method to the RTSP protocol. The key frame image data of the specified preview time requested by the preview command can be transmitted through the target streaming media connection session used by the first media stream data for preview display. Without affecting the established media stream, the client can perform picture preview. Moreover, since the existing target streaming media connection session is used, only the performance overhead of sending image data is consumed to implement preview of the image at the specified preview time during video playback, reducing the unnecessary LAPI message interaction overhead or unnecessary cache overhead on the device side brought by adding the image preview function.

[0083] Figure 5The figure is a schematic structural diagram of a data processing device provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the situation of video playback and image preview on a client side. The data processing device can be implemented in the form of software and / or hardware, and is generally integrated on any electronic device with network communication functions. The electronic device can be a mobile terminal, a PC, a server, etc.

[0084] As Figure 5 shown, the data processing device according to the embodiment of the present invention may include: a connection module 510, a first display module 520, and a second display module 530. Among them:

[0085] The connection module 510 is configured to establish a target streaming media connection session between the client and the server for supporting the transmission of streaming media data and image data based on the Real-Time Streaming Protocol;

[0086] The first display module 520 is configured to obtain first media stream data sent by the server through the target streaming media connection session, and perform data display in a first display area based on the first media stream data. The first media stream data is formed by a part of the streaming media data requested by the server in the target streaming media data based on the target play command of the client;

[0087] The second display module 530 is configured to, when obtaining the first media stream data sent by the server, obtain second media stream data sent by the server through the target streaming media connection session, and perform data display in a second display area based on the second media stream data. The second media stream data is formed by key frame image data of a specified preview time requested by the server in the target streaming media data based on the target preview command of the client.

[0088] Based on the technical solution of the above embodiment, optionally, establishing a target streaming media connection session between the client and the server for supporting the transmission of streaming media data and image data includes:

[0089] Sending a streaming media description command to the server, so that the server sends target streaming media session description information to the client. The media type in the target streaming media session description information is for supporting the transmission of streaming media data and image data, and the target streaming media description information indicates the target streaming media data to be transmitted in the server;

[0090] According to the received target streaming media session description information, sending a streaming media connection establishment command to the server to establish a target streaming media connection session for transmitting the target streaming media data of the server between the client and the server;

[0091] Establishing a target streaming media connection session between the client and the server after receiving the streaming media connection response message sent by the server.

[0092] Based on the technical solutions of the above embodiments, optionally, obtaining the first media stream data sent by the server through the target streaming media connection session includes:

[0093] Sending a target play command to the server through the target streaming media connection session to instruct the server to encode and encapsulate the partial streaming media data requested by the target play command in the target streaming media data based on the Real-Time Transport Protocol to obtain the first media stream data; the play command is an original command in the Real-Time Streaming Protocol for instructing the server to start sending streaming media data to the client based on the transmission mechanism corresponding to the streaming media connection session;

[0094] Receiving the first media stream data sent by the server through the target streaming media connection session.

[0095] Based on the technical solutions of the above embodiments, optionally, displaying data in the first display area based on the first media stream data includes:

[0096] Decoding the first media stream data and performing rendering display in the first display area based on the decoded first media data to play back the streaming media data corresponding to the first media stream data.

[0097] Based on the technical solutions of the above embodiments, optionally, obtaining the second media stream data sent by the server through the target streaming media connection session includes:

[0098] Sending a target preview command to the server through the target streaming media connection session to instruct the server to encode and encapsulate the key frame image data of the specified preview time requested by the target preview command in the target streaming media data based on the Real-Time Transport Protocol to obtain the second media stream data; the preview command is a newly added command in the Real-Time Streaming Protocol for instructing the server to start sending the image data of the specified preview time to the client based on the transmission mechanism corresponding to the streaming media connection session;

[0099] Receiving the second media stream data sent by the server through the target streaming media connection session.

[0100] Based on the technical solutions of the above embodiments, optionally, displaying data in the second display area based on the second media stream data includes:

[0101] Decoding the second media stream data and performing rendering display in the second display area based on the decoded key frame image data of the specified preview time to preview the key frame image data corresponding to the second media stream data.

[0102] Based on the technical solutions of the above embodiments, optionally, the second display area is a preview display window that pops up after selecting a specified preview time on the time axis corresponding to the target streaming media data and is used to render and display the image data of the specified preview time.

[0103] Based on the technical solutions of the above embodiments, optionally, sending a target preview command to the server through the target streaming media connection session includes:

[0104] In response to a trigger operation on the time axis corresponding to the target streaming media data, determining the specified preview time corresponding to the trigger operation;

[0105] Sending a target preview command to the server through the target streaming media connection session for previewing and displaying the key frame image of the specified preview time in the target streaming media data.

[0106] Based on the technical solutions of the above embodiments, optionally, the device further includes:

[0107] Detecting the type of the media stream sent by the server through the target streaming media connection session, and the type of the media stream can be distinguished based on the encoding and encapsulation format used by the media stream.

[0108] The technical solutions provided by the embodiments of the present invention support adding new media types through extended configuration of the Real-Time Streaming Protocol (RTSP). By adding the OPTIONS request method in the RTSP, the key frame image data of the specified preview time requested through the preview command can be transmitted through the target streaming media connection session adopted by the first media stream data for preview display. Without affecting the established media stream, the client can perform picture preview. Moreover, since the existing target streaming media connection session is used, only the performance overhead of sending image data is required to achieve preview of the image at the specified preview time during video playback, reducing the unnecessary LAPI message interaction overhead or unnecessary cache overhead on the device side brought by adding the image preview function.

[0109] The data processing device provided by the embodiments of the present invention can execute the data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data processing method.

[0110] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiments of the present invention.

[0111] Figure 6The following is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Refer to Figure 6 , which shows a schematic structural diagram of an electronic device (such as a Figure 6 terminal device or server in) 600 suitable for implementing the embodiments of the present invention. The terminal device in the embodiments of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0112] As Figure 6 shown, the electronic device 600 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which may perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 into the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The editing / output (I / O) interface 605 is also connected to the bus 604.

[0113] Generally, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or wirelesly to exchange data. Although Figure 6 shows the electronic device 600 having various devices, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0114] In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for performing the data processing method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the data processing method of the embodiment of the present invention are executed.

[0115] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are for illustrative purposes only and are not used to limit the scope of these messages or information.

[0116] The electronic device provided in the embodiment of the present invention and the data processing method provided in the above embodiment belong to the same inventive concept. Technical details not described in detail in this embodiment can be referred to in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0117] The embodiment of the present invention provides a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, the data processing method provided in the above embodiment is implemented.

[0118] It should be noted that the above-mentioned computer-readable medium of the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present invention, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0119] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0120] The above-mentioned computer-readable medium can be included in the above-mentioned electronic device; it can also exist separately without being assembled into the electronic device.

[0121] The above computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: establish a target streaming media connection session between the client and the server for supporting the transmission of streaming media data and image data based on the Real-Time Streaming Protocol; obtain the first media stream data sent by the server through the target streaming media connection session, and perform data display in a first display area based on the first media stream data, where the first media stream data is formed by a part of the streaming media data requested by the server from the target streaming media data based on the target play command of the client; when obtaining the first media stream data sent by the server, obtain the second media stream data sent by the server through the target streaming media connection session, and perform data display in a second display area based on the second media stream data, where the second media stream data is formed by the key frame image data of the specified preview time requested by the server from the target streaming media data based on the target preview command of the client.

[0122] Computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The above programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet).

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0124] The units described in the embodiments of the present invention can be implemented in software or in hardware. Among them, the name of the unit does not constitute a limitation on the unit itself in some cases. For example, the first acquisition unit can also be described as "the unit for acquiring at least two Internet protocol addresses".

[0125] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chip (SOCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0126] In the context of the present invention, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM or Flash memory), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] The above description is only a preferred embodiment of the present invention and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present invention.

[0128] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0129] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms for implementing the claims.

Claims

1. A data processing method, characterized in that, applied to a client, the method includes: Establishing a target streaming media connection session between the client and the server for supporting the transmission of streaming media data and image data based on the Real-Time Streaming Protocol; Obtaining first media stream data sent by the server through the target streaming media connection session, and performing data display in a first display area based on the first media stream data, where the first media stream data is formed by a part of the streaming media data requested by the server in the target streaming media data based on the target play command of the client; When obtaining the first media stream data sent by the server, obtaining second media stream data sent by the server through the target streaming media connection session, and performing data display in a second display area based on the second media stream data, where the second media stream data is formed by key frame image data of a specified preview time requested by the server in the target streaming media data based on the target preview command of the client.

2. The method according to claim 1, characterized in that, Establishing a target streaming media connection session between the client and the server for supporting the transmission of streaming media data and image data includes: Sending a streaming media description command to the server to cause the server to send target streaming media session description information to the client, where the media type in the target streaming media session description information is for supporting the transmission of streaming media data and image data, and the target streaming media description information indicates the target streaming media data to be transmitted in the server; According to the received target streaming media session description information, sending a streaming media connection establishment command to the server to establish a target streaming media connection session between the client and the server for transmitting the target streaming media data of the server; Establishing a target streaming media connection session between the client and the server after receiving the streaming media connection response message sent by the server.

3. The method according to claim 1, characterized in that, Obtaining second media stream data sent by the server through the target streaming media connection session includes: Sending a target preview command to the server through the target streaming media connection session to instruct the server to encode and encapsulate key frame image data of a specified preview time requested by the target preview command in the target streaming media data based on the Real-Time Transport Protocol to obtain second media stream data; the preview command is a newly added command in the Real-Time Streaming Protocol for instructing the server to start sending image data of a specified preview time to the client using the transmission mechanism corresponding to the streaming media connection session; Receiving second media stream data sent by the server through the target streaming media connection session.

4. The method according to claim 1 or 3, characterized in that, Performing data display in a second display area based on the second media stream data includes: Decoding the second media stream data, and performing rendering display in the second display area based on the decoded key frame image data of the specified preview time to preview the key frame image data corresponding to the second media stream data.

5. The method according to claim 4, characterized in that, The second display area is a preview display window that pops up after specifying a preview time is triggered on the timeline corresponding to the target streaming media data and is used to render and display the image data of the specified preview time.

6. The method according to claim 3, wherein, sending a target preview command to the server through the target streaming media connection session includes: responding to a trigger operation on the timeline corresponding to the target streaming media data, determining the specified preview time corresponding to the trigger operation; sending, through the target streaming media connection session, a target preview command for previewing and displaying a key frame image of the specified preview time in the target streaming media data to the server.

7. The method according to claim 1, wherein, the method further includes: detecting the type of the media stream sent by the server through the target streaming media connection session, and the type of the media stream can be distinguished based on the encoding and encapsulation format used by the media stream.

8. A data processing device, wherein, configured on the client side, the device includes: a connection module, configured to establish a target streaming media connection session between the client and the server based on the real-time streaming protocol for supporting the transmission of streaming media data and image data; a first display module, configured to obtain the first media stream data sent by the server through the target streaming media connection session, and perform data display in the first display area based on the first media stream data, where the first media stream data is formed by a part of the streaming media data requested by the server in the target streaming media data based on the target play command of the client; a second display module, configured to obtain the second media stream data sent by the server through the target streaming media connection session when obtaining the first media stream data sent by the server, and perform data display in the second display area based on the second media stream data, where the second media stream data is formed by the key frame image data of the specified preview time requested by the server in the target streaming media data based on the target preview command of the client.

9. An electronic device, wherein, the electronic device includes: one or more processors; a storage device, configured to store one or more programs, when the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the data processing method according to any one of claims 1-7.

10. A storage medium containing computer-executable instructions, wherein, the computer-executable instructions are used to execute the data processing method according to any one of claims 1-7 when executed by a computer processor.

Images