WebSocket-based media data management and control method

Abstract

The invention provides a media data management and control method based on WebSocket. The method comprises the following steps: connecting with a WebSocket media server of a front-end webpage, carrying out identity authentication, providing a WebSocket communication interface, carrying out interaction by using an internally agreed WebSocket signaling, calling a gRPC interface corresponding to an internal service to realize requested operation, and sending a corresponding RTSP signaling to a media forwarding server to control a media data stream state; based on a request about a media data stream, the WebSocket media server directly forwards the media data stream, repeated pulling of the same media data is avoided, before the media data stream is sent, the WebSocket server converts a media data stream code into a format, so that a front-end page can directly play the media data stream, a plurality of modules subordinate to the WebSocket media server are mutually called to realize the functions, and the media data stream can be directly played by the WebSocket media server. And the media data processing convenience and security are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a media data management and control method based on WebSocket. Background Art

[0002] When the front-end web page needs to play media data, it communicates directly with the media forwarding server, first obtains the URL of the corresponding media data stream through TCP, and then uses the URL to pull the media data stream through RTSP and play it. And when the front-end web page needs to control the device, it communicates directly with the device management server and uses TCP to send commands to control the device behavior. However, the latest mainstream browsers no longer support native playback of RTSP streams, and users cannot directly play media data pulled from the media forwarding server; and the front-end web page also needs to communicate directly with multiple internal services, which will expose more ports and cannot improve the convenience and security of media data playback processing. Summary of the invention

[0003] The purpose of the present invention is to provide a media data management and control method based on WebSocket, which is connected and authenticated with the WebSocket media server of the front-end web page, and provides a WebSocket communication interface, so as to interact with the internally agreed WebSocket signaling, thereby calling the gRPC interface of the corresponding internal service to implement the requested operation and sending the corresponding RTSP signaling to the media forwarding server to control the media data stream state; based on the request for the media data stream, the WebSocket media server directly executes the forwarding process to avoid repeatedly pulling the same media data. Before sending the media data stream, the WebSocket server converts the media data stream code into FMP4 format, so that the front-end page can be played directly. It uses the WebSocket module, MediaRPC module, and Media module under the WebSocket media server to call each other to realize the above functions, thereby improving the convenience and security of media data processing.

[0004] The present invention is achieved through the following technical solutions:

[0005] The media data management and control method based on WebSocket includes:

[0006] Identify the login request initiated by the SaaS end to the WebSocket media server, and determine whether the SaaS end has successfully logged into the WebSocket media server;

[0007] When the SaaS end successfully logs into the WebSocket media server, the signaling type information received by the WebSocket module of the WebSocket media server is determined;

[0008] Based on the signaling type information, determine to call the MediaRPC module and / or Media module of the WebSocket media server to perform corresponding internal service operations and / or media data operations; wherein the internal service operations include pan / tilt / lens control; the media data operations include media data preview, video file query, media data playback or media data fast / slow playback.

[0009] Optionally, identifying a login request initiated by the SaaS end to the WebSocket media server, and determining whether the SaaS end successfully logs into the WebSocket media server, includes:

[0010] Transmitting the login request initiated by the SaaS end to the WebSocket media server to the WebSocket module of the WebSocket media server, thereby identifying the login request; if the login request is successfully identified, returning the existing token to the SaaS end, and determining that the SaaS end has successfully logged into the WebSocket media server;

[0011] If the login request fails to be identified, the WebSocket module is instructed to perform Java token verification; if the verified Java token is illegal, a login failure result is returned to the SaaS end; if the verified Java token is legal, a random token is generated and saved, and then the random token is returned to the SaaS end.

[0012] Optionally, when the SaaS end successfully logs into the WebSocket media server, the signaling type information received by the WebSocket module of the WebSocket media server is determined, including:

[0013] When the SaaS end successfully logs in to the WebSocket media server, the code content of the signaling received by the WebSocket module is identified to determine whether the signaling received by the WebSocket module is non-media operation related signaling or media operation related signaling.

[0014] Optionally, based on the signaling type information, determining to call a MediaRPC module or a Media module of the WebSocket media server to perform corresponding internal service operations and / or media data operations includes:

[0015] When the signaling received by the WebSocket module is non-media operation related signaling, the MediaRPC module and the Media module of the WebSocket media server are called to perform internal service operations;

[0016] When the signaling received by the WebSocket module is a media operation related signaling, the MediaRPC module and / or the Media module of the WebSocket media server is called to perform media data operations.

[0017] Optionally, calling the MediaRPC module of the WebSocket media server to perform internal service operations of PTZ / lens control includes:

[0018] The pan / tilt / lens control instruction contained in the non-media operation related signaling is sent to the pan / tilt / lens control interface of the WebSocket module, and the pan / tilt / lens control instruction is parsed; if the data parsing fails, a pan / tilt control failure result is returned to the SaaS end; if the data parsing is successful, the corresponding pan / tilt / lens control interface is called to the MediaRPC module to perform the pan / tilt / lens control operation, and a pan / tilt control success result is returned to the SaaS end through the WebSocket module.

[0019] Optionally, calling the MediaRPC module and the Media module of the WebSocket media server to perform media data operations for media data preview includes:

[0020] Based on the media operation related signaling, a preview opening request is initiated to open the preview interface in the WebSocket module; a token check is performed based on the preview interface of the WebSocket module; if the checked token is illegal, a media data preview failure result is returned to the SaaS end; if the checked token is legal, a preview interface is opened in the Media module of the WebSocket media server to perform a ctrl class instruction check; if the check shows that a ctrl class instruction exists, it is determined whether the Media module is playing media data; if the check shows that a ctrl class instruction does not exist, a new ctrl class instruction is generated, and then it is determined whether the Media module is playing media data; if media data is being played, a media data preview failure result is returned to the SaaS end; if media data is not being played, a preview interface is opened in the MediaRPC module of the WebSocket media server;

[0021] Based on the preview interface of the MediaRPC module, the corresponding preview media data is retrieved, and after the preview media data is fMP4 encoded, it is previewed and played in the Media module.

[0022] Optionally, calling the MediaRPC module of the WebSocket media server to perform a media data operation for video file query includes:

[0023] Based on the media operation related signaling, a video file query request is initiated to open the video file query interface in the WebSocket module; a token check is performed based on the video file query interface; if the checked token is illegal, a video file query failure result is returned to the SaaS end; if the detected token is legal, the MediaRPC module of the WebSocket media server opens the video file query interface to retrieve the corresponding video file, and returns the video file query success result to the SaaS end.

[0024] Optionally, calling the MediaRPC module and the Media module of the WebSocket media server to perform media data operations for media data playback includes:

[0025] Based on the media operation related signaling, a playback request is initiated to open the playback interface in the WebSocket module; a token check is performed based on the playback interface of the WebSocket module; if the checked token is illegal, a media data playback failure result is returned to the SaaS end; if the checked token is legal, the playback interface is opened in the Media module of the WebSocket media server to perform a ctrl class instruction check; if the check shows that a ctrl class instruction exists, it is determined whether the Media module is playing media data; if the check shows that a ctrl class instruction does not exist, a new ctrl class instruction is generated, and then it is determined whether the Media module is playing media data; if media data is being played, a media data playback failure result is returned to the SaaS end; if media data is not being played, the playback interface is opened in the MediaRPC module of the WebSocket media server;

[0026] Based on the playback interface of the MediaRPC module, the corresponding playback media data is retrieved, and the playback media data is fMP4 encoded according to the playback speed and then played back in the Media module.

[0027] Optionally, calling the Media module of the WebSocket media server to perform a media data operation of fast / slow playback of media data includes:

[0028] Based on the media operation related signaling, a media data fast / slow playback request is initiated to open a fast / slow playback interface in the WebSocket module; and media data is parsed based on the fast / slow playback interface to determine whether the media data parsing is successful; if unsuccessful, a media data fast / slow playback failure result is returned to the SaaS end; if successful, a fast / slow playback interface is opened in the Media module of the WebSocket media server; based on the fast / slow playback interface of the Media module, the media data fast / slow playback speed is set to perform media data operations for fast / slow playback of the media data.

[0029] Optionally, based on the signaling type information, determining to call a MediaRPC module and / or a Media module of the WebSocket media server to perform corresponding internal service operations and media data operations includes:

[0030] Step S1, using the following formula (1), based on the signaling type information, perform corresponding internal service operations and media data operations,

[0031]

[0032] In the above formula (1), C represents the sum of internal service operations; M represents the product of media data operations; S j represents the jth value of the signaling type, where S(j)=S1 represents PTZ control, S(j)=S2 represents lens control, S(j)=S3 represents video playback, and S(j)=S4 represents file query; f j (k) represents the influence degree value of the control operation corresponding to the jth value of the signaling type; m represents the total number of signaling; gk(V) represents the complexity value of the media data operation;

[0033] Step S2, using the following formula (2), based on the internal service operation and the media data operation, the internal service operation processing efficiency is obtained:

[0034]

[0035] In the above formula (2), E represents the internal service operation processing efficiency; T i represents the processing time of the i-th internal service operation; R i Indicates the priority of the i-th internal service operation; N j represents the number of devices controlled by the jth internal service operation; Q j represents the complexity coefficient of the jth internal service operation; n represents the number of signaling related to the internal service operation;

[0036] Step S3, using the following formula (3), based on the internal service operation processing efficiency, determine the execution type of the internal service operation,

[0037]

[0038] In the above formula (3), CON i represents the execution type determination value of the i-th internal service operation; P i represents the operation parameters of the i-th internal service operation; W i represents the weight coefficient of the i-th internal service operation; D i represents the standard deviation of the i-th internal service operation; S i represents the standard value corresponding to the standard deviation of the i-th internal service operation; H i represents the regulation index of the ith internal service operation.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The WebSocket-based media data management and control method provided in the present application is connected to and authenticated by the WebSocket media server of the front-end web page, and provides a WebSocket communication interface, so as to interact using the internally agreed WebSocket signaling, thereby calling the gRPC interface of the corresponding internal service to implement the requested operation and sending the corresponding RTSP signaling to the media forwarding server to control the media data stream status; based on the request for the media data stream, the WebSocket media server directly executes the forwarding process to avoid repeatedly pulling the same media data. Before sending the media data stream, the WebSocket server converts the media data stream code into FMP4 format so that the front-end page can be played directly. It uses the WebSocket module, MediaRPC module, and Media module under the WebSocket media server to call each other to implement the above functions, thereby improving the convenience and security of media data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0042] Figure 1 A schematic flow chart of a WebSocket-based media data management and control method provided by the present invention.

[0043] Figure 2 A schematic diagram of the architecture of a WebSocket media server corresponding to the WebSocket-based media data management and control method provided by the present invention.

[0044] Figure 3 A schematic diagram of a general workflow of a WebSocket media server corresponding to the WebSocket-based media data management and control method provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0047] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] See also Figure 1 As shown, an embodiment of the present application provides a method for managing and controlling media data based on WebSocket. The method for managing and controlling media data based on WebSocket includes:

[0049] Identify the login request initiated by the SaaS end to the WebSocket media server, and determine whether the SaaS end has successfully logged into the WebSocket media server;

[0050] When the SaaS client successfully logs into the WebSocket media server, the signaling type information received by the WebSocket module of the WebSocket media server is determined;

[0051] Based on the signaling type information, determine to call the MediaRPC module and / or Media module of the WebSocket media server to perform corresponding internal service operations and / or media data operations; wherein the internal service operations include pan / tilt / lens control; the media data operations include media data preview, video file query, media data playback or media data fast / slow playback.

[0052] The beneficial effects of the above embodiments are as follows: the WebSocket-based media data management and control method is connected and authenticated with the WebSocket media server of the front-end web page, and provides a WebSocket communication interface, thereby using the internally agreed WebSocket signaling to interact, thereby calling the gRPC interface of the corresponding internal service to implement the requested operation and sending the corresponding RTSP signaling to the media forwarding server to control the media data stream status; based on the request for the media data stream, the WebSocket media server directly executes the forwarding process to avoid repeatedly pulling the same media data. Before sending the media data stream, the WebSocket server converts the media data stream code into FMP4 format, so that the front-end page can be played directly. It uses the WebSocket module, MediaRPC module, and Media module under the WebSocket media server to call each other to implement the above functions, thereby improving the convenience and security of media data processing.

[0053] In another embodiment, identifying a login request initiated by a SaaS end to a WebSocket media server and determining whether the SaaS end successfully logs into the WebSocket media server includes:

[0054] The login request initiated by the SaaS end to the WebSocket media server is transmitted to the WebSocket module of the WebSocket media server to identify the login request; if the login request is successfully identified, the existing token is returned to the SaaS end, and it is determined that the SaaS end has successfully logged into the WebSocket media server;

[0055] If the login request fails to be recognized, the WebSocket module is instructed to perform Java token verification; if the verified Java token is illegal, a login failure result is returned to the SaaS end; if the verified Java token is legal, a random token is generated and saved, and then the random token is returned to the SaaS end.

[0056] The beneficial effects of the above embodiments are that the WebSocket media server includes three functional modules: the WebSocket module, the MediaRPC module and the Media module. These three functional modules can directly call each other. Among them, the WebSocket module is used to manage the WebSocket connection and identity authentication with the front-end page, and provide a WebSocket communication interface. The WebSocket communication interface uses the internally agreed WebSocket signaling for command interaction, and calls the rRPC interface of the corresponding internal service according to the agreed WebSocket signaling to implement the requested operation. The MediaRPC module is used to manage the RTSP connection with the media forwarding server, and sends the corresponding RTSP signaling to the media forwarding server according to the agreed WebSocket signaling to control the status of the media data stream. The Media module is used to provide forwarding and transcoding functions. For requests for the same media data stream, the WebSocket media server will directly execute the forwarding process to avoid repeatedly pulling the same media data stream. Before sending the media data stream, the WebSocket media server will transcode the media data stream into FMP4 format to allow the front-end web page to play directly. Among them, the architecture and general workflow of the WebSocket media server corresponding to the WebSocket-based media data management and control method of the present application are respectively as follows: Figure 2 and 3 As shown in the figure, the corresponding WebSocket module, MediaRPC module and Media module work interaction and their respective work functions can be seen from Figure 2-3It is obtained directly, and no repetitive description is made here. Specifically, when the login request initiated by the SaaS end to the WebSocket media server is transmitted to the WebSocket media server, the WebSocket module of the WebSocket media server identifies the login request, and performs corresponding token verification, thereby ensuring accurate and efficient verification of the login request of the SaaS end, and improving the login security of the WebSocket media server. In addition, if the login request fails to be identified, the WebSocket module is instructed to perform Java token verification; if the verified Java token is illegal, the login failure result is returned to the SaaS end; if the verified Java token is legal, a random token is generated and saved, and then the random token is returned to the SaaS end, which can ensure that the SaaS end obtains the result of whether it has successfully logged in in time. In addition, when the SaaS end re-initiates a login request, the WebSocket module will delete its own token and related information to avoid the original token and related information from interfering with the subsequent new login request authentication.

[0057] In another embodiment, when the SaaS end successfully logs into the WebSocket media server, the signaling type information received by the WebSocket module of the WebSocket media server is determined, including:

[0058] When the SaaS end successfully logs into the WebSocket media server, the code content of the signaling received by the WebSocket module is identified to determine whether the signaling received by the WebSocket module is non-media operation related signaling or media operation related signaling.

[0059] The beneficial effect of the above embodiment is that the WebSocket media server can perform media operations associated with media data and non-media operations not associated with media data. For this reason, when the SaaS end successfully logs in to the WebSocket media server, the code content of the signaling received by the WebSocket module is identified, so that the operation action that needs to be executed corresponding to the signaling can be accurately identified, and it is determined whether the signaling received by the WebSocket module is non-media operation-related signaling or media operation-related signaling, so as to differentiate and execute subsequent non-media operations and media operations, thereby improving the operational reliability of the WebSocket media server.

[0060] In another embodiment, based on the signaling type information, determining to call the MediaRPC module and / or the Media module of the WebSocket media server to perform corresponding internal service operations or media data operations includes:

[0061] When the signaling received by the WebSocket module is non-media operation related signaling, the MediaRPC module and the Media module of the WebSocket media server are called to perform internal service operations;

[0062] When the signaling received by the WebSocket module is a media operation-related signaling, the MediaRPC module and / or the Media module of the WebSocket media server is called to perform media data operations.

[0063] The beneficial effect of the above embodiment is that when the signaling received by the WebSocket module is a signaling related to non-media operations, the MediaRPC module and the Media module of the WebSocket media server are called to perform internal service operations, so that both the MediaRPC module and the Media module are called at the same time to perform internal service operations not associated with media data, thereby improving the execution reliability of non-media operations. When the signaling received by the WebSocket module is a signaling related to media operations, the MediaRPC module and / or the Media module of the WebSocket media server are called to perform media data operations, so that any one or both of the MediaRPC module and the Media module can be called to perform media data operations, thereby improving the execution reliability of media operations.

[0064] In another embodiment, calling the MediaRPC module of the WebSocket media server to perform internal service operations of PTZ / lens control includes:

[0065] The pan / tilt / lens control instruction contained in the non-media operation related signaling is sent to the pan / tilt / lens control interface of the WebSocket module, and the pan / tilt / lens control instruction is parsed; if the data parsing fails, the pan / tilt control failure result is returned to the SaaS end; if the data parsing is successful, the corresponding pan / tilt / lens control interface is called to the MediaRPC module to perform the pan / tilt / lens control operation, and the pan / tilt control success result is returned to the SaaS end through the WebSocket module.

[0066] The beneficial effect of the above embodiment is that the pan / tilt / lens is used to adjust the image shooting action, which does not involve the processing of media data. At this time, the WebSocket media server performs non-media operations accordingly. For this purpose, the pan / tilt / lens control instruction contained in the non-media operation related signaling is sent to the pan / tilt / lens control interface of the WebSocket module, and the pan / tilt / lens control instruction is parsed; if the data parsing fails, the pan / tilt control failure result is returned to the SaaS end; if the data parsing is successful, the corresponding pan / tilt / lens control interface is called to the MediaRPC module to perform the pan / tilt / lens control operation, wherein the pan / tilt / lens control operation can be but not limited to pan / tilt posture adjustment and / or lens zoom adjustment, thereby improving the operation reliability of the pan / tilt / lens.

[0067] In another embodiment, calling the MediaRPC module and the Media module of the WebSocket media server to perform media data operations for media data preview includes:

[0068] Based on the media operation related signaling, a preview request is initiated to open the preview interface in the WebSocket module; a token check is performed based on the preview interface of the WebSocket module; if the token is invalid, a media data preview failure result is returned to the SaaS end; if the token is valid, a preview interface is opened in the Media module of the WebSocket media server to perform a ctrl-type instruction check; if the ctrl-type instruction exists, it is determined whether the Media module is playing media data; if the ctrl-type instruction does not exist, a new ctrl-type instruction is generated, and then it is determined whether the Media module is playing media data; if the media data is being played, a media data preview failure result is returned to the SaaS end; if the media data is not being played, a preview interface is opened in the MediaRPC module of the WebSocket media server;

[0069] Based on the preview interface of the MediaRPC module, the corresponding preview media data is retrieved, and after the preview media data is fMP4 encoded, it is previewed and played in the Media module.

[0070] The beneficial effect of the above embodiment is that when the media operation-related signaling requires media data preview, the preview interface is first opened inside the WebSocket module. At this time, a token check is performed on the preview interface of the WebSocket module to further determine whether the preview interface is opened in the Media module and the MeidaRPC module, so as to accurately retrieve the corresponding media data that needs to be previewed inside the MeidaRPC module and convert the retrieved media data into fMP4 format in the Media module, thereby ensuring that the Media module performs effective and stable media data preview and playback.

[0071] In another embodiment, calling the MediaRPC module of the WebSocket media server to perform a media data operation for video file query includes:

[0072] Based on the media operation related signaling, a video file query request is initiated to open the video file query interface in the WebSocket module; a token check is performed based on the video file query interface; if the checked token is illegal, a video file query failure result is returned to the SaaS end; if the detected token is legal, the MediaRPC module of the WebSocket media server opens the video file query interface to retrieve the corresponding video file, and returns the video file query success result to the SaaS end.

[0073] The beneficial effect of the above embodiment is that when the media operation-related signaling needs to query the video file, the video file query interface is first opened inside the WebSocket module. At this time, a token check is performed on the video file query interface of the WebSocket module to further determine whether to open the video file query interface in the MediaRPC module, so as to accurately retrieve the video file inside the MediaRPC module, ensuring that the queried video file is returned to the SaaS end in a timely manner.

[0074] In another embodiment, calling the MediaRPC module and the Media module of the WebSocket media server to perform media data operations for media data playback includes:

[0075] Based on the media operation related signaling, initiate a playback request to open the playback interface in the WebSocket module; perform a token check based on the playback interface of the WebSocket module; if the checked token is illegal, return a media data playback failure result to the SaaS end; if the checked token is legal, open the playback interface in the Media module of the WebSocket media server to perform a ctrl class instruction check; if the check shows that a ctrl class instruction exists, determine whether the Media module is playing media data; if the check shows that a ctrl class instruction does not exist, generate a new ctrl class instruction, and then determine whether the Media module is playing media data; if media data is being played, return a media data playback failure result to the SaaS end; if media data is not being played, open the playback interface in the MediaRPC module of the WebSocket media server;

[0076] Based on the playback interface of the MediaRPC module, the corresponding playback media data is retrieved, and the playback media data is encoded in fMP4 according to the playback speed and then played back in the Media module.

[0077] The beneficial effect of the above embodiment is that when the media operation-related signaling requires media data playback, the playback interface is first opened in the WebSocket module. At this time, a token check is performed on the playback interface of the WebSocket module to further determine whether the playback interface is opened in the Media module and the MediaRPC module, so as to accurately retrieve the corresponding media data that needs to be played back inside the MediaRPC module and convert the retrieved media data into fMP4 format, thereby ensuring that the Media module performs effective and stable media data playback operations.

[0078] In another embodiment, calling the Media module of the WebSocket media server to perform a media data operation of fast / slow playback of media data includes:

[0079] Based on the media operation related signaling, a media data fast / slow playback request is initiated to open the fast / slow playback interface in the WebSocket module; and media data is parsed based on the fast / slow playback interface to determine whether the media data parsing is successful; if unsuccessful, a media data fast / slow playback failure result is returned to the SaaS end; if successful, a fast / slow playback interface is opened in the Media module of the WebSocket media server; based on the fast / slow playback interface of the Media module, the media data fast / slow playback speed is set to perform media data operations for fast / slow playback of the media data.

[0080] The beneficial effect of the above embodiment is that when the media operation-related signaling needs to make a request for fast / slow playback of media data, the fast / slow playback interface is first opened in the WebSocket module. At this time, the media data received by the fast / slow playback interface is parsed and processed, and whether the media data parsing result is successful is determined, so that the Media module can obtain the data basis for fast / slow playback, and the fast / slow playback speed of the media data is set based on the fast / slow playback interface of the Media module, so as to perform media data operations for fast / slow playback of the media data, which can ensure that the Media module can accurately perform fast / slow playback at a predetermined playback speed.

[0081] In another embodiment, based on the signaling type information, determining to call the MediaRPC module and / or the Media module of the WebSocket media server to perform corresponding internal service operations and media data operations includes:

[0082] Step S1, using the following formula (1), based on the signaling type information, perform corresponding internal service operations and media data operations.

[0083]

[0084] In the above formula (1), C represents the sum of internal service operations; M represents the product of media data operations; S j represents the jth value of the signaling type, where S(j)=S1 represents PTZ control, S(j)=S2 represents lens control, S(j)=S3 represents video playback, and S(j)=S4 represents file query; f j (k) represents the influence degree value of the control operation corresponding to the jth value of the signaling type; m represents the total number of signaling; gk(V) represents the complexity value of the media data operation;

[0085] Step S2, using the following formula (2), based on the internal service operation and the media data operation, the internal service operation processing efficiency is obtained:

[0086]

[0087] In the above formula (2), E represents the internal service operation processing efficiency; T i represents the processing time of the i-th internal service operation; R i Indicates the priority of the i-th internal service operation; N j represents the number of devices controlled by the jth internal service operation; Q j represents the complexity coefficient of the jth internal service operation; n represents the number of signaling related to the internal service operation;

[0088] Step S3, using the following formula (3), based on the internal service operation processing efficiency, determine the execution type of the internal service operation.

[0089]

[0090] In the above formula (3), CON i represents the execution type determination value of the i-th internal service operation, where the execution type of the internal service operation may include controlling the pan / tilt rotation angle or the lens focal length; i represents the operation parameters of the i-th internal service operation, where the operation parameters include the control of the pan / tilt rotation angle or the lens focal length; W i represents the weight coefficient of the i-th internal service operation; D i represents the standard deviation of the i-th internal service operation; S i represents the standard value corresponding to the standard deviation of the i-th internal service operation; H i represents the regulation index of the ith internal service operation.

[0091] The beneficial effects of the above embodiment are as follows: using the above formula (1), based on the signaling type information, corresponding internal service operations and media data operations are performed, the selection of operation types is determined, and the operation types are mapped by signaling types and influencing factors; then using the above formula (2), based on internal service operations and media data operations, the internal service operation processing efficiency is obtained, and by combining the time, priority, quantity and complexity of each operation, the evaluation of data operation efficiency is further improved; finally, using the above formula (3), based on the internal service operation processing efficiency, the execution type of the internal service operation is determined, and then through the relationship between the deviation and the standard value, precise control of the pan / tilt and lens is achieved.

[0092] In general, the WebSocket-based media data management and control method connects and authenticates the WebSocket media server of the front-end web page, and provides a WebSocket communication interface, thereby using the internally agreed WebSocket signaling to interact, thereby calling the gRPC interface of the corresponding internal service to implement the requested operation and sending the corresponding RTSP signaling to the media forwarding server to control the media data stream status; based on the request for the media data stream, the WebSocket media server directly executes the forwarding process to avoid repeatedly pulling the same media data. Before sending the media data stream, the WebSocket server converts the media data stream code into FMP4 format, so that the front-end page can be played directly. It uses the WebSocket module, MediaRPC module, and Media module under the WebSocket media server to call each other to implement the above functions, thereby improving the convenience and security of media data processing.

[0093] The above is only a specific implementation of the present invention, and any other improvements made based on the concept of the present invention are considered to be within the protection scope of the present invention.

Claims

1. A media data management and control method based on WebSocket, characterized in that: include: Identify the login request initiated by the SaaS end to the WebSocket media server, and determine whether the SaaS end has successfully logged into the WebSocket media server; When the SaaS end successfully logs into the WebSocket media server, the signaling type information received by the WebSocket module of the WebSocket media server is determined; Based on the signaling type information, determine to call the MediaRPC module and / or Media module of the WebSocket media server to perform corresponding internal service operations and / or media data operations; wherein the internal service operations include pan / tilt / lens control; the media data operations include media data preview, video file query, media data playback or media data fast / slow playback.

2. The WebSocket-based media data management and control method according to claim 1, characterized in that: Identifying a login request initiated by the SaaS end to the WebSocket media server and determining whether the SaaS end successfully logs into the WebSocket media server includes: Transmitting the login request initiated by the SaaS end to the WebSocket media server to the WebSocket module of the WebSocket media server, thereby identifying the login request; if the login request is successfully identified, returning the existing token to the SaaS end, and determining that the SaaS end has successfully logged into the WebSocket media server; If the login request fails to be identified, the WebSocket module is instructed to perform Java token verification; if the verified Java token is illegal, a login failure result is returned to the SaaS end; if the verified Java token is legal, a random token is generated and saved, and then the random token is returned to the SaaS end.

3. The WebSocket-based media data management and control method according to claim 1, wherein: When the SaaS client successfully logs in to the WebSocket media server, the signaling type information received by the WebSocket module of the WebSocket media server is determined, including: When the SaaS end successfully logs in to the WebSocket media server, the code content of the signaling received by the WebSocket module is identified to determine whether the signaling received by the WebSocket module is non-media operation related signaling or media operation related signaling.

4. The WebSocket-based media data management and control method according to claim 3, characterized in that: Based on the signaling type information, determining to call the MediaRPC module and / or the Media module of the WebSocket media server to perform corresponding internal service operations or media data operations includes: When the signaling received by the WebSocket module is non-media operation related signaling, the MediaRPC module and the Media module of the WebSocket media server are called to perform internal service operations; When the signaling received by the WebSocket module is a media operation related signaling, the MediaRPC module and / or the Media module of the WebSocket media server is called to perform media data operations.

5. The WebSocket-based media data management and control method according to claim 4, characterized in that: Calling the MediaRPC module of the WebSocket media server to perform internal service operations for PTZ / lens control includes: The pan / tilt / lens control instruction contained in the non-media operation related signaling is sent to the pan / tilt / lens control interface of the WebSocket module, and the pan / tilt / lens control instruction is parsed; if the data parsing fails, a pan / tilt control failure result is returned to the SaaS end; if the data parsing is successful, the corresponding pan / tilt / lens control interface is called to the MediaRPC module to perform the pan / tilt / lens control operation, and a pan / tilt control success result is returned to the SaaS end through the WebSocket module.

6. The WebSocket-based media data management and control system according to claim 4, characterized in that: Calling the MediaRPC module and the Media module of the WebSocket media server to perform media data operations for media data preview includes: Based on the media operation related signaling, a preview opening request is initiated to open the preview interface in the WebSocket module; a token check is performed based on the preview interface of the WebSocket module; if the checked token is illegal, a media data preview failure result is returned to the SaaS end; if the checked token is legal, a preview interface is opened in the Media module of the WebSocket media server to perform a ctrl class instruction check; if the check shows that a ctrl class instruction exists, it is determined whether the Media module is playing media data; if the check shows that a ctrl class instruction does not exist, a new ctrl class instruction is generated, and then it is determined whether the Media module is playing media data; if media data is being played, a media data preview failure result is returned to the SaaS end; if media data is not being played, a preview interface is opened in the MediaRPC module of the WebSocket media server; Based on the preview interface of the MediaRPC module, the corresponding preview media data is retrieved, and after the preview media data is fMP4 encoded, it is previewed and played in the Media module.

7. The WebSocket-based media data management and control system according to claim 4, characterized in that: Calling the MediaRPC module of the WebSocket media server to perform media data operations for video file query includes: Based on the media operation related signaling, a video file query request is initiated to open the video file query interface in the WebSocket module; a token check is performed based on the video file query interface; if the checked token is illegal, a video file query failure result is returned to the SaaS end; if the detected token is legal, the MediaRPC module of the WebSocket media server opens the video file query interface to retrieve the corresponding video file, and returns the video file query success result to the SaaS end.

8. The WebSocket-based media data management and control system according to claim 4, characterized in that: Calling the MediaRPC module and the Media module of the WebSocket media server to perform media data operations for media data playback includes: Based on the media operation related signaling, a playback request is initiated to open the playback interface in the WebSocket module; a token check is performed based on the playback interface of the WebSocket module; if the checked token is illegal, a media data playback failure result is returned to the SaaS end; if the checked token is legal, the playback interface is opened in the Media module of the WebSocket media server to perform a ctrl class instruction check; if the check shows that a ctrl class instruction exists, it is determined whether the Media module is playing media data; if the check shows that a ctrl class instruction does not exist, a new ctrl class instruction is generated, and then it is determined whether the Media module is playing media data; if media data is being played, a media data playback failure result is returned to the SaaS end; if media data is not being played, the playback interface is opened in the MediaRPC module of the WebSocket media server; Based on the playback interface of the MediaRPC module, the corresponding playback media data is retrieved, and the playback media data is fMP4 encoded according to the playback speed and then played back in the Media module.

9. The WebSocket-based media data management and control system according to claim 4, characterized in that: Calling the Media module of the WebSocket media server to perform media data operations for fast / slow playback of media data includes: Based on the media operation related signaling, a media data fast / slow playback request is initiated to open a fast / slow playback interface in the WebSocket module; and media data is parsed based on the fast / slow playback interface to determine whether the media data parsing is successful; if unsuccessful, a media data fast / slow playback failure result is returned to the SaaS end; if successful, a fast / slow playback interface is opened in the Media module of the WebSocket media server; based on the fast / slow playback interface of the Media module, the media data fast / slow playback speed is set to perform media data operations for fast / slow playback of the media data.

10. The WebSocket-based media data management and control system according to claim 1, characterized in that: Based on the signaling type information, determining to call the MediaRPC module and / or the Media module of the WebSocket media server to perform corresponding internal service operations and media data operations includes: Step S1, using the following formula (1), based on the signaling type information, perform corresponding internal service operations and media data operations, In the above formula (1), C represents the sum of internal service operations; M represents the product of media data operations; S j represents the jth value of the signaling type, where S(j)=S1 represents PTZ control, S(j)=S2 represents lens control, S(j)=S3 represents video playback, and S(j)=S4 represents file query; f j (k) represents the influence degree value of the control operation corresponding to the jth value of the signaling type; m represents the total number of signaling; gk(V) represents the complexity value of the media data operation; Step S2, using the following formula (2), based on the internal service operation and the media data operation, the internal service operation processing efficiency is obtained: In the above formula (2), E represents the internal service operation processing efficiency; T i represents the processing time of the i-th internal service operation; R i Indicates the priority of the i-th internal service operation; N j represents the number of devices controlled by the jth internal service operation; Q j represents the complexity coefficient of the jth internal service operation; n represents the number of signaling related to the internal service operation; Step S3, using the following formula (3), based on the internal service operation processing efficiency, determine the execution type of the internal service operation, In the above formula (3), CON i represents the execution type determination value of the i-th internal service operation; P i represents the operation parameters of the i-th internal service operation; W i represents the weight coefficient of the i-th internal service operation; D i represents the standard deviation of the i-th internal service operation; S i represents the standard value corresponding to the standard deviation of the i-th internal service operation; H i represents the regulation index of the ith internal service operation.

Images