A multimedia data processing method and system based on cloud storage

Abstract

This application provides a multimedia data processing method and system based on cloud storage. For different multimedia data types, the system adaptively monitors the business connection status through a business connection status monitoring module. If the connection is normal, the multimedia business data is saved to the cloud. If the connection is broken and data transmission is incomplete, the audio and video analysis module authenticates, parses the protocol, and processes the transmitted data, extracts the valid parts, and saves them to object storage. After the connection is re-established, the processed results are provided to the user terminal through an event notification mechanism. This method effectively ensures data integrity without increasing the workload on the business side and significantly reduces the resource overhead on the user side.

Description

Technical Field

[0001] This application relates to the field of large-scale distributed block storage technology, specifically to a multimedia data processing method and system based on cloud storage. Background Technology

[0002] With the rapid development of network infrastructure and multimedia technology, the volume of multimedia data has grown exponentially in recent years. As a major component of multimedia data, surveillance video recording poses a significant challenge to corresponding storage technologies. Object storage, with its flexibility and flatness, allows for the storage of massive amounts of data through scalability and enables efficient access and management of files via browsers, HTTP RESTful APIs, and other methods, effectively meeting the needs of video recording and storage in the cloud era. Taking a video surveillance scenario as an example, cameras upload generated data through the object storage's putObject interface. As an interface, putObject returns success if the data uploaded by the camera has been saved, and failure if it has not. If the camera's upload request fails due to network disconnection, although some data has been transmitted to the object storage service, this data is not saved, resulting in incomplete video recording. Traditionally, if a network problem occurs when the camera calls the upload interface to upload an object, the entire upload request fails, and object storage discards the received data, affecting the integrity of the video recording.

[0003] To improve the integrity of video recordings, the industry has proposed several optimization methods:

[0004] (1) Add an SD card to the client for caching, but this requires additional hardware costs, and some older cameras cannot be modified;

[0005] (2) Use small slice upload method, such as changing the original 1 minute slice object to 4 seconds slice object. Originally, a failed request would lose 1 minute of video, but after changing to small slice, only 4 seconds of video will be lost. However, this will increase the number of files that the business platform management and object storage need to manage by 15 times, and the index storage of the business platform and object storage will increase significantly.

[0006] (3) Use the small slice appending method. For example, append once every 4 seconds, and after appending for 1 minute, switch to another file for appending. Compared with small slice uploading, this method can reduce the number of video file indexes maintained by the business platform, but the number of additional small slice indexes that object storage needs to maintain is not reduced.

[0007] Since disconnection scenarios do not occur frequently, typically less than 1%, using the three optimization methods mentioned earlier would require significant resource overhead. Summary of the Invention

[0008] To address the aforementioned technical problems, this application provides a multimedia data processing method and system based on cloud storage, which can adaptively monitor the service connection status through a service connection status monitoring module for different multimedia data types. The technical solution adopted in this application is as follows:

[0009] A multimedia data processing system based on cloud storage, comprising an object storage access layer module, an object storage persistence layer module, a service connection status monitoring module, an audio and video analysis and extraction module, and an event notification module;

[0010] The object storage access layer module is connected to the event notification module, the service connection status monitoring module, the audio and video analysis and extraction module, and the object storage persistence layer module, respectively. It receives client requests forwarded by the service connection status monitoring module, authenticates the client requests, receives multimedia data from the client after successful authentication, forwards the multimedia data to the audio and video analysis and extraction module, and forwards the analysis results returned by the audio and video analysis and extraction module to the object storage persistence layer module for storage.

[0011] The service connection status monitoring module is connected to the client and the object storage access layer module respectively, monitors the service connection status in real time, and adaptively adjusts the monitoring method according to the data type, forwarding the received client requests to the object storage access layer module.

[0012] The audio and video analysis and extraction module receives multimedia data from the client forwarded by the object storage access layer module, parses and preprocesses the multimedia data, extracts the valid content and corresponding metadata information from the multimedia data, and returns the analysis results to the object storage access layer module.

[0013] The object storage persistence layer module is used to persist object storage data and indexes;

[0014] The event notification module is connected to the business platform and the object storage access layer module, respectively. It is used to receive, save, and distribute messages from the object storage access layer module, and notify the business platform of the time range corresponding to the multimedia data for storage.

[0015] Furthermore, the business platform is connected to the event notification module for multimedia data recording management.

[0016] Furthermore, the multimedia data recording management includes device management, user management, and maintaining the correspondence between the timeline and the object storage file name.

[0017] Furthermore, in the audio and video analysis and extraction module, the parsing and preprocessing includes protocol parsing and data decapsulation to obtain multimedia stream data; decoding the decapsulated data and extracting multimedia metadata; and preprocessing the extracted multimedia metadata.

[0018] Furthermore, the preprocessing includes deduplication, metadata field splitting, and metadata field format normalization, and finally, the effective content is extracted based on the metadata information.

[0019] Furthermore, in the business connection status monitoring module, the monitoring method is adaptively adjusted according to the data type, including: monitoring the request messages sent by the client, identifying the request messages, obtaining the type field and code field values ​​of the request messages, and adaptively selecting the monitoring method based on the type field and code field.

[0020] Furthermore, the business connection status monitoring module monitors the business connection status, including: detecting dead connections based on preset parameters. For a TCP connection that has been established through a three-way handshake, if there is no data packet transmission between the two parties within the keep-alive time, the keep-alive function is enabled, and the cloud storage end will send a keep-alive data heartbeat packet. If no response is received, the keep-alive data heartbeat packet is sent again at each keep-alive time interval, and so on for N times. If no response is received, an RST packet is sent to close the connection. If a response is received, the timer is reset to zero.

[0021] A multimedia data processing method based on cloud storage, applied to the aforementioned multimedia data processing system, includes the following steps:

[0022] Step 1: Monitor the service connection status with the client through the service connection status monitoring module;

[0023] Step 2: Select different processing flows based on the monitoring results. If the business connection is normal, execute the normal processing flow and save the multimedia data to the cloud storage through the object storage persistence layer module. If the business connection is abnormal, execute the abnormal processing flow and save the valid part of the multimedia data to the cloud storage through the object storage persistence layer module.

[0024] Step 3: Re-establish the business connection status monitoring module and the client, and use the event notification module to feed back the processing results of the selected process to the client.

[0025] Furthermore, the normal processing procedure includes the following steps:

[0026] Step 101: The client generates an access request carrying multimedia data and uploads it to the object storage access layer module;

[0027] Step 102: The object storage access layer module receives the access request, authenticates the access request, and then fully receives the multimedia data carried in the access request.

[0028] Step 103: The object storage access layer module forwards the multimedia data to the audio and video analysis and extraction module for parsing and preprocessing;

[0029] Step 104: The object storage access layer module receives the analysis results of the multimedia data from the audio and video analysis and extraction module, and calls the interface of the object storage persistence layer module to save the analysis results;

[0030] Step 105: The object storage access layer module packages the correspondence between multimedia time periods and file names into a message and forwards it to the event notification module;

[0031] Step 106: After receiving the message, the event notification module saves it and distributes the message to the business platform, and the business platform records the multimedia data index.

[0032] Furthermore, the exception handling process includes the following steps:

[0033] Step 201: The client generates an access request carrying multimedia data and uploads it to the object storage access layer module;

[0034] Step 202: The object storage access layer module receives the access request, authenticates the access request, and then receives the multimedia data carried in the access request. If the network connection is disconnected before the multimedia data is fully received, the connection between the object storage access layer module and the client times out and the connection between the object storage access layer module and the client is disconnected.

[0035] Step 203: The object storage access layer module sends the received multimedia data to the audio and video analysis and extraction module for analysis, extracts the playable valid data and corresponding metadata from the multimedia data, and calls the interface of the object storage persistence layer module to save the analysis results.

[0036] Step 204: The object storage access layer module forwards the notification message to the event notification module. After receiving the message, the event notification module saves it and distributes the message to the business platform. The business platform then updates the correspondence between the time period and the file name of the multimedia data.

[0037] The embodiments of this application can achieve the following technical effects: it is applicable to a variety of multimedia data types, and can adaptively monitor the service connection status according to the data type, which can more effectively determine the disconnection and improve the real-time performance and effectiveness of network status monitoring; it can analyze audio and video data to obtain valid images, audio and valid short videos from incomplete data received. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is an architecture diagram of a multimedia data processing system based on object cloud storage.

[0040] Figure 2 This is a schematic diagram of the normal processing flow;

[0041] Figure 3 This is a schematic diagram of the exception handling process. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] The multimedia data processing method and system based on cloud storage provided in this application are applied to a general object storage system architecture on the cloud storage side. When the connection is broken and data transmission is not completed, the audio and video analysis module authenticates, parses the protocol and processes the transmitted data, extracts the valid part and saves it to the object storage. After the connection is re-established, the processed results are provided to the user end through an event notification mechanism. Without increasing the extra work on the business side, the integrity of the data is more effectively guaranteed and the resource consumption on the user side is greatly reduced. Figure 1 This is an architecture diagram of a multimedia data processing system based on object cloud storage. The system includes an object storage access layer module, an object storage persistence layer module, a business connection status monitoring module, an audio and video analysis and extraction module, and an event notification module.

[0044] The object storage access layer module is connected to the event notification module, the service connection status monitoring module, the audio and video analysis and extraction module, and the object storage persistence layer module, respectively. It receives client requests forwarded by the service connection status monitoring module, authenticates the client requests, receives multimedia data from the client after successful authentication, forwards the multimedia data to the audio and video analysis and extraction module, and forwards the analysis results returned by the audio and video analysis and extraction module to the object storage persistence layer module for storage.

[0045] The service connection status monitoring module is connected to the client and the object storage access layer module respectively, monitors the service connection status in real time, and adaptively adjusts the monitoring method according to the data type, forwarding the received client requests to the object storage access layer module.

[0046] The audio and video analysis and extraction module receives multimedia data from the client forwarded by the object storage access layer module, parses and preprocesses the multimedia data, extracts the valid content and corresponding metadata information from the multimedia data, and returns the analysis results to the object storage access layer module.

[0047] The object storage persistence layer module is used to persist object storage data and indexes;

[0048] The event notification module is connected to the business platform and the object storage access layer module, respectively. It is used to receive, save and distribute messages from the object storage access layer module, and notify the business platform of the time range corresponding to the multimedia data for storage.

[0049] The business platform is connected to the event notification module and is used for multimedia data recording management;

[0050] The multimedia data recording management includes device management, user management, and maintaining the correspondence between the timeline and the object storage file name.

[0051] In the audio and video analysis and extraction module, the parsing and preprocessing include protocol parsing and data decapsulation, such as protocol layers including HTTP, RTMP, etc., to obtain multimedia stream data, such as audio stream data or video stream data; decoding the decapsulated data and extracting multimedia metadata, for example, taking video stream as an example, reading video stream (video data stream) information, extracting video metadata, setting a unique identifier for each piece of video metadata, and storing the video metadata, its unique identifier, and the data source identifier together; preprocessing the extracted multimedia metadata, including deduplication, metadata field splitting, and metadata field format normalization; finally, extracting valid content based on the metadata information, such as images, audio, and valid short videos.

[0052] In the business connection status monitoring module, the monitoring method is adaptively adjusted according to the data type, including: monitoring the request messages sent by the client, identifying the request messages, obtaining the type field and code field values ​​of the request messages, and adaptively selecting the monitoring method based on the type field and code field.

[0053] The business connection status monitoring module monitors the business connection status, including: detecting dead connections based on preset parameters. For a TCP connection that has been established through a three-way handshake, if there is no data packet transmission between the two parties within the keep-alive time, the keep-alive function is enabled, and the cloud storage end will send a keep-alive data heartbeat packet. If no response is received, the keep-alive data heartbeat packet is sent again at each keep-alive time interval, and so on for N times. If no response is received, an RST packet is sent to close the connection. If a response is received, the timer is reset to zero.

[0054] The preset parameters include the number of probe retries; if the number of probe retries is exceeded, the connection is closed; and the probe heartbeat interval. If no data transmission is performed after the probe heartbeat interval, a probe live message is sent. If no reply is received, the message is resent according to the probe heartbeat interval.

[0055] Different multimedia data have varying transmission times and file sizes. Different clients also have different network connection states. The method described above can adaptively select different probing methods for different data transmission types, such as choosing the number of probing retries and the heartbeat interval. It also performs a preliminary analysis of the network status at the individual long-connection level, dynamically adjusting the probing method and interval.

[0056] exist Figure 1 In the middle, the client module (camera) generates the data that needs to be saved and transmits the data to object storage through the interface.

[0057] The module includes an object storage access layer, an object storage persistence layer, a service connection status monitoring module, an audio / video analysis and extraction module, and an event notification module.

[0058] The multimedia data processing method based on cloud storage in this application includes the following steps:

[0059] Step 1: Monitor the service connection status with the client through the service connection status monitoring module;

[0060] Step 2: Select different processing flows based on the monitoring results. If the business connection is normal, execute the normal processing flow and save the multimedia data to the cloud storage through the object storage persistence layer module. If the business connection is abnormal, execute the abnormal processing flow and save the valid part of the multimedia data to the cloud storage through the object storage persistence layer module.

[0061] Step 3: Re-establish the business connection status monitoring module and the client, and use the event notification module to feed back the processing results of the selected process to the client.

[0062] The schematic diagram of the normal processing flow in the above method is as follows: Figure 2 As shown, the normal processing flow includes the following steps:

[0063] Step 101: The client generates an access request carrying multimedia data and uploads it to the object storage access layer module;

[0064] Step 102: The object storage access layer module receives the access request, authenticates the access request, and then fully receives the multimedia data carried in the access request.

[0065] Step 103: The object storage access layer module forwards the multimedia data to the audio and video analysis and extraction module for parsing and preprocessing;

[0066] Step 104: The object storage access layer module receives the analysis results of the multimedia data from the audio and video analysis and extraction module, and calls the interface of the object storage persistence layer module to save the analysis results;

[0067] Step 105: The object storage access layer module packages the correspondence between multimedia time periods and file names into a message and forwards it to the event notification module;

[0068] Step 106: After receiving the message, the event notification module saves it and distributes the message to the business platform, and the business platform records the multimedia data index.

[0069] In the above method, a schematic diagram of the exception handling process is shown below. Figure 3 As shown, the exception handling process includes the following steps:

[0070] Step 201: The client generates an access request carrying multimedia data and uploads it to the object storage access layer module;

[0071] Step 202: The object storage access layer module receives the access request, authenticates the access request, and then receives the multimedia data carried in the access request. If the network connection is disconnected before the multimedia data is fully received, the connection between the object storage access layer module and the client times out and the connection between the object storage access layer module and the client is disconnected.

[0072] Step 203: The object storage access layer module sends the received multimedia data to the audio and video analysis and extraction module for analysis, extracts the playable valid data and corresponding metadata from the multimedia data, and calls the interface of the object storage persistence layer module to save the analysis results.

[0073] Step 204: The object storage access layer module forwards the notification message to the event notification module. After receiving the message, the event notification module saves it and distributes the message to the business platform. The business platform then updates the correspondence between the time period and the file name of the multimedia data.

[0074] In step 202, since the connection between the object storage access layer module and the client is broken, the client will not receive a successful reception response from the object storage access layer module. When the connection between the object storage access layer module and the client expires, the client notifies the business platform to update the index.

[0075] In summary, the method and system of this application, on the cloud storage side, analyze and process incomplete multimedia data that has not yet been fully transmitted, extracting valid content (such as video screenshots and audio extraction) and corresponding metadata information (such as start and end times), and saving it to object storage, thereby reducing data loss rate and ensuring data integrity. Adaptive monitoring of business connection status involves protocol parsing of multimedia data and adaptively adjusting monitoring strategies for different data types. This monitoring can dynamically and automatically adjust for individual connections. The extracted metadata is processed, and this application can automatically adjust event notification rules according to different data types, notifying the user side and completing the linkage between the mobile terminal and the server. Furthermore, the method and system of this application only require additional analysis, processing, and message notification when a disconnection occurs, significantly reducing resource overhead on the business side while maintaining the same integrity rate. Through optimization of the cloud storage side, this application simplifies the data processing logic on the user side, eliminating the need for methods such as slicing and appending uploads to improve data integrity. Compared to user-side append-only uploads, this method reduces the number of video file indexes maintained by the business platform, and simplifies user-side processing. Compared to user-side small-slice uploads, this patent significantly reduces the index storage required by the business platform and object storage.

[0076] While specific embodiments of this application have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this application is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this application, but all such changes and modifications fall within the scope of protection of this application.

Claims

1. A multimedia data processing system based on cloud storage, characterized in that, The system includes an object storage access layer module, an object storage persistence layer module, a service connection status monitoring module, an audio and video analysis and extraction module, and an event notification module; The object storage access layer module is connected to the event notification module, the service connection status monitoring module, the audio and video analysis and extraction module, and the object storage persistence layer module, respectively. It receives client requests forwarded by the service connection status monitoring module, authenticates the client requests, receives multimedia data from the client after successful authentication, forwards the multimedia data to the audio and video analysis and extraction module, and forwards the analysis results returned by the audio and video analysis and extraction module to the object storage persistence layer module for storage. The service connection status monitoring module is connected to the client and the object storage access layer module respectively, monitors the service connection status in real time, and adaptively adjusts the monitoring method according to the data type, forwarding the received client requests to the object storage access layer module. The audio and video analysis and extraction module receives multimedia data from the client forwarded by the object storage access layer module, parses and preprocesses the multimedia data, extracts the valid content and corresponding metadata information from the multimedia data, and returns the analysis results to the object storage access layer module. The object storage persistence layer module is used to persist object storage data and indexes; The event notification module is connected to the business platform and the object storage access layer module, respectively. It is used to receive, save and distribute messages from the object storage access layer module, and notify the business platform of the time range corresponding to the multimedia data for storage. The service connection status monitoring module monitors the service connection status with the client. Different processing flows are selected based on the monitoring results. If the business connection is normal, the normal processing flow is executed, and the multimedia data is saved to the cloud storage through the object storage persistence layer module. If the business connection is abnormal, the abnormal processing flow is executed, and the valid part of the multimedia data is saved to the cloud storage through the object storage persistence layer module.

2. The system according to claim 1, characterized in that, The business platform is connected to the event notification module and is used for multimedia data recording management.

3. The system according to claim 2, characterized in that, The multimedia data recording management includes device management, user management, and maintaining the correspondence between the timeline and the object storage file name.

4. The system according to claim 1, characterized in that, In the audio and video analysis and extraction module, the parsing and preprocessing include protocol parsing and data decapsulation to obtain multimedia stream data; decoding the decapsulated data and extracting multimedia metadata; and preprocessing the extracted multimedia metadata.

5. The system according to claim 4, characterized in that, The preprocessing includes deduplication, metadata field splitting, and metadata field format normalization. Finally, the effective content is extracted based on the metadata information.

6. The system according to claim 1, characterized in that, In the business connection status monitoring module, the monitoring method is adaptively adjusted according to the data type, including: monitoring the request messages sent by the client, identifying the request messages, obtaining the type field and code field values ​​of the request messages, and adaptively selecting the monitoring method based on the type field and code field.

7. The system according to claim 1, characterized in that, The business connection status monitoring module monitors the business connection status, including: detecting dead connections based on preset parameters. For a TCP connection that has been established through a three-way handshake, if there is no data packet transmission between the two parties within the keep-alive time, the keep-alive function is enabled, and the cloud storage end will send a keep-alive data heartbeat packet. If no response is received, the keep-alive data heartbeat packet is sent again at each keep-alive time interval, and so on for N times. If no response is received, an RST packet is sent to close the connection. If a response is received, the timer is reset to zero.

8. A multimedia data processing method based on cloud storage, wherein the method is applied to the multimedia data processing system as described in any one of claims 1 to 7, characterized in that, The method includes the following steps: Step 1: Monitor the service connection status with the client through the service connection status monitoring module; Step 2: Select different processing flows based on the monitoring results. If the business connection is normal, execute the normal processing flow and save the multimedia data to the cloud storage through the object storage persistence layer module. If the business connection fails, the exception handling process will be executed, and the valid part of the multimedia data will be saved to cloud storage through the object storage persistence layer module. Step 3: Re-establish the business connection status monitoring module and the client, and use the event notification module to feed back the processing results of the selected process to the client.

9. The method according to claim 8, characterized in that, The normal processing procedure includes the following steps: Step 101: The client generates an access request carrying multimedia data and uploads it to the object storage access layer module; Step 102: The object storage access layer module receives the access request, authenticates the access request, and then fully receives the multimedia data carried in the access request. Step 103: The object storage access layer module forwards the multimedia data to the audio and video analysis and extraction module for parsing and preprocessing; Step 104: The object storage access layer module receives the analysis results of the multimedia data from the audio and video analysis and extraction module, and calls the interface of the object storage persistence layer module to save the analysis results; Step 105: The object storage access layer module packages the correspondence between multimedia time periods and file names into a message and forwards it to the event notification module; Step 106: After receiving the message, the event notification module saves it and distributes the message to the business platform, and the business platform records the multimedia data index.

10. The method according to claim 8, characterized in that, The exception handling process includes the following steps: Step 201: The client generates an access request carrying multimedia data and uploads it to the object storage access layer module; Step 202: The object storage access layer module receives the access request, authenticates the access request, and then receives the multimedia data carried in the access request. If the network connection is disconnected before the multimedia data is fully received, the connection between the object storage access layer module and the client times out and the connection between the object storage access layer module and the client is disconnected. Step 203: The object storage access layer module sends the received multimedia data to the audio and video analysis and extraction module for analysis, extracts the playable valid data and corresponding metadata from the multimedia data, and calls the interface of the object storage persistence layer module to save the analysis results. Step 204: The object storage access layer module forwards the notification message to the event notification module. After receiving the message, the event notification module saves it and distributes the message to the business platform. The business platform then updates the correspondence between the time period and the file name of the multimedia data.

Images