Method and system for directly storing audio and video streams to cloud

Abstract

The invention relates to a method and system for directly storing audio and video streams to a cloud end, and the method comprises the steps: enabling an audio and video collection terminal to carry out the coding and packaging of a video stream according to a GOP as a unit, and generating a transmission stream segment; establishing a data channel with a cloud server based on a link parameter provided by the signaling server, and uploading the transport stream fragment to the cloud server; after an upload success response is received, deleting the local transport stream segments, and continuously transmitting the remaining transport stream segments until a termination signal is detected; in response to the termination signal, sending a termination request to the cloud server; and based on the termination request, the cloud server performs merging operation on all the transport stream fragments which are accumulatively uploaded at this time, and a complete transport stream file is generated and stored. Through application of the method and the device, the problem that the memory pressure of the audio and video acquisition terminal and the cloud management efficiency cannot be considered at the same time due to the fixed slice duration is solved, and reasonable optimization of the local memory and cloud resources is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video monitoring, in particular to a method and system for streaming audio and video directly to the cloud. BACKGROUND

[0002] In the existing network camera audio and video cloud recording scheme, the common practice is that the camera performs slicing processing (for example, generates a TS file) on the audio and video data, and these slice files are then uploaded to the cloud storage. Due to the small memory of the embedded camera, in order to balance the system performance and resource management, the length of the slice is usually set to 10 to 30 seconds. However, this traditional scheme has the following problems:

[0003] 1. High local memory occupation: If the slice time is too long, the camera needs to cache a large TS file locally, which causes high memory pressure and affects the device running efficiency.

[0004] 2. Too many small files in the cloud: If the slice time is too short, the cloud storage will generate a large number of small files, increasing the burden of database index information and affecting the system performance and management efficiency.

[0005] 3. Upload delay and resource waste: The fixed-length slicing mechanism lacks flexibility and is difficult to adapt to the recording needs in different scenarios.

[0006] At present, there is no effective solution to the dilemma caused by the fixed slice time in the related technology: if the slice is too long, the memory pressure of the camera is high, and if the slice is too short, the cloud is flooded with small files. SUMMARY

[0007] The embodiments of the present application provide a method and system for streaming audio and video directly to the cloud, an electronic device and a storage medium, to at least solve the problem that the memory pressure of the camera and the management efficiency of the cloud cannot be considered in the related technology due to the fixed slice length.

[0008] In a first aspect, the embodiments of the present application provide a method for streaming audio and video directly to the cloud, executed by an audio and video acquisition terminal, comprising:

[0009] encoding and packaging the obtained current video stream in units of GOPs to generate a current transport stream segment;

[0010] requesting a cloud server connection parameter from a signaling server and establishing a data channel with the cloud server;

[0011] uploading the current transport stream segment to the cloud server through the data channel, deleting the current transport stream segment buffered in the audio and video collection terminal after receiving a successful upload response from the cloud server, and continuously transmitting the remaining transport stream segments until a termination signal is detected; the remaining transport stream segments refer to segments generated by encoding and encapsulating the remaining video stream; the remaining video stream is a video stream other than the current video stream in the obtained real-time video stream;

[0012] In response to the termination signal, a termination request is sent to the cloud server; based on the termination request, the cloud server performs a merging operation on all transport stream segments accumulated this time to generate a complete transport stream file and store it.

[0013] In some embodiments, the termination signal includes a first termination signal; the continuous transmission of the remaining transport stream segments until the termination signal is detected includes:

[0014] continuously transmitting the remaining transport stream segments, and when it is detected that the audio and video collection terminal uploads through the data channel for a long time and the cumulative time reaches a first time threshold, the first termination signal is generated, and in response to the first termination signal, a first termination request is sent to the cloud server.

[0015] In some embodiments, the termination signal includes a second termination signal; the continuous transmission of the remaining transport stream segments until the termination signal is detected includes:

[0016] continuously transmitting the remaining transport stream segments, and when a special event alarm state is detected, the second termination signal is generated, and in response to the second termination signal, a second termination request is sent to the cloud server.

[0017] In some embodiments, when the special event is detected, for the abnormal video stream of the special event, the method further includes:

[0018] down-regulating the first time threshold to a target value to generate a second time threshold;

[0019] encoding and encapsulating the abnormal video stream in GOP units to generate an abnormal transport stream segment;

[0020] requesting new connection parameters of the cloud server from the signaling server, establishing a new data channel with the cloud server; uploading the abnormal transport stream segment to the cloud server through the new data channel;

[0021] invoke the second time threshold; send a third termination signal to the cloud server based on the second time threshold; wherein the cloud server is configured to perform a merging operation on all the abnormal transport stream segments accumulated this time based on the third termination signal, generate a new complete transport stream file and store it;

[0022] when it is detected that the special event alarm state ends, resume invoking the first time threshold.

[0023] In some embodiments, the method further comprises:

[0024] maintain the data channel established when the current transport stream segment is first uploaded;

[0025] multiplex the same data channel, and continuously perform real-time upload processing on the remaining transport stream segments through the data channel.

[0026] In some embodiments, after the encoding and encapsulation of the acquired video stream in units of GOPs to generate transport stream segments, the method further comprises:

[0027] If it is detected that the number of transport stream segments buffered in the audio and video acquisition terminal exceeds a maximum number threshold, the earliest generated transport stream segment is deleted in the order of the generation time of the buffered transport stream segments.

[0028] In a second aspect, the embodiments of the present application provide a system for streaming audio and video directly to the cloud, comprising an audio and video acquisition terminal, a cloud server and a signaling server;

[0029] The signaling server is configured to establish a data channel between the audio and video acquisition terminal and the cloud server.

[0030] The audio and video acquisition terminal is configured to perform the method for streaming audio and video directly to the cloud as described in the first aspect above.

[0031] The cloud server is further configured to perform a merging operation on all the transport stream segments accumulated this time based on the termination request, generate a complete transport stream file and store it.

[0032] In some embodiments, the cloud server is further configured to verify the time sequence continuity of the transport stream segments and check the integrity of the frame header of each transport stream segment.

[0033] In some embodiments, the system further comprises a user terminal.

[0034] The user terminal is configured to send a video playback request to the signaling server.

[0035] The signaling server is further configured to parse the video playback request and obtain time range information carried in the video playback request.

[0036] The signaling server is further configured to retrieve the index information, determine target index information matched with the time range information, obtain a storage path in the target index information, and return the storage path to the user terminal; the storage path points to the complete transport stream file stored in the cloud server.

[0037] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method for streaming audio and video directly to the cloud according to the first aspect.

[0038] In a fourth aspect, a storage medium is provided, which stores a computer program executable by a processor to implement the method for streaming audio and video directly to the cloud according to the first aspect.

[0039] Compared with the related art, the method for streaming audio and video directly to the cloud according to the embodiments of the present application encodes and encapsulates the video stream into transport stream segments with GOP as a unit, and combines the technical solution of “generating, uploading, and deleting simultaneously”, thereby solving the problems of local memory overflow and cloud resource fragmentation, effectively managing the local memory, significantly reducing the local memory occupation of the audio and video acquisition terminal, reducing the requirement on the device hardware, and improving the overall operation efficiency of the device; meanwhile, the use of the cloud resources is optimized, the cloud storage resources are reasonably managed, the continuity of video playback is ensured, the management complexity caused by too many small files is avoided, and the overall efficiency of system operation is improved.

[0040] Details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects, and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings illustrated herein are used to provide further understanding of the present application, constitute a part of the present application, and the illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 is a hardware structure block diagram of a terminal according to the method for streaming audio and video directly to the cloud;

[0043] Figure 2 is a flowchart of the method for streaming audio and video directly to the cloud according to the embodiments of the present application;

[0044] Figure 3 FIG. 9 is a flowchart illustrating another method for streaming and storing audio and video directly to the cloud according to an embodiment of the present application;

[0045] Figure 4 FIG. 10 is a block diagram illustrating a method for streaming and storing audio and video directly to the cloud according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application and should not be used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of the present application. In addition, it should be understood that although the efforts made in this development process can be complex and lengthy, some design, manufacture or production changes made on the basis of the technical content disclosed in the present application by those of ordinary skill in the art related to the content disclosed in the present application are only routine technical means and should not be understood as insufficient disclosure of the present application.

[0047] In the present application, the phrase "embodiments" means that the specific features, structures or characteristics described in conjunction with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0048] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. Unless otherwise defined, the terms "one" and "a" or "an" used in the present application shall not be construed as being limited to a single element, but instead are used to represent one or more elements. The terms "including," "comprising," "having," and variations thereof in the present application are meant to encompass the elements listed thereafter and variations thereof, unless otherwise specified. For example, a process, method, system, product, or apparatus that comprises a list of steps or units (elements) is not limited to those steps or units which are recited, but can also include additional steps or units that are not expressly listed or inherent to such process, method, system, product, or apparatus. The terms "connected," "coupled," and variations thereof in the present application are not limited to direct connections, but can also include indirect connections, unless otherwise specified. The term "plurality" in the present application means two or more. The term "and / or" in the present application describes associated objects, which means that there can be three relationships: for example, "A and / or B" can mean that A exists alone, A and B exist together, or B exists alone. The terms "first," "second," "third," and the like in the present application are only used to distinguish similar objects, and do not represent a specific order.

[0049] The method embodiments provided by the present embodiment can be executed in a terminal, a computer, or a similar computing device. Taking a terminal as an example, Figure 1 is a hardware structure block diagram of a terminal according to the method of streaming audio and video to the cloud. As shown in Figure 1 , the terminal can include one or more (only one is shown in Figure 1 ) processor 102 (the processor 102 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal can also include a transmission device 106 for communication functions and an input and output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, and does not limit the structure of the terminal. For example, the terminal can include more or fewer components than those shown in Figure 1 , or have a different configuration than that shown in Figure 1 .

[0050] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the method of streaming audio and video directly to the cloud in the embodiments of the present application. The processor 102 performs various functional applications and data processing by running the computer programs stored in the memory 104, that is, implements the above method. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the terminal through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0051] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network can include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module used to communicate with the Internet in a wireless manner.

[0052] The embodiments of the present application provide a method of streaming audio and video directly to the cloud, Figure 2 is a flowchart of the method of streaming audio and video directly to the cloud according to the embodiments of the present application, as shown in Figure 2 The flowchart includes the following steps:

[0053] In step S201, the current video stream obtained is encoded and packaged in units of GOPs to generate a current transport stream segment.

[0054] The audio and video acquisition terminal captures a real-time video stream, and encodes and packages the current video stream obtained in units of GOPs to generate a current transport stream segment. The audio and video acquisition terminal refers to a device capable of generating / acquiring an audio and video stream and encoding and packaging the audio and video stream, including a camera, a smart device camera, an image acquisition card, an AR / VR acquisition terminal, an AI edge computing box, a UAV aerial terminal, a medical imaging terminal, and a vehicle-mounted intelligent recorder.

[0055] GOP (Group of Pictures) refers to a sequence of continuous video frames compressed and encoded according to a specific structure. A typical GOP consists of three types of frames: I frames (Intra-coded pictures), P frames (Predictive-coded pictures), and B frames (Bidirectionally predictive-coded pictures). Each GOP must start with an I frame, containing several P frames and B frames. I frames, also known as key frames, are compressed using only the spatial redundancy of the current frame itself and do not reference other frames. P frames are compressed using both the temporal redundancy (motion changes) between the reference frame and the current frame, and the spatial redundancy of the current frame. B frames are compressed using the bidirectional temporal redundancy between the front and back frames (which can be I frames or P frames).

[0056] Since GOPs start with I frames, which are independently decodable without relying on any other frames, and due to the internal closed-loop dependency within GOPs, P frames within a GOP only reference previous I / P frames within the same GOP (e.g., a P frame references a previous I frame or P frame), and B frames may reference both the previous and subsequent frames within the same GOP (e.g., a B frame references a previous P frame and a subsequent I frame), but never rely on frames from the next GOP. Therefore, all reference frames required for a complete GOP are within the GOP itself, and no external frames are needed for decoding. Each GOP has independent decoding capabilities, so the current transport stream segment generated in GOP units also has independent decoding capabilities. The current transport stream segment refers to an independent data unit generated by the audio and video acquisition terminal after decoding and compressing the video stream, commonly represented by TS files (extension.ts).

[0057] Specifically, taking a camera as an example, the camera captures images according to predetermined encoding parameters (such as H.264 / H.265 encoding standards) and generates TS files in GOP units (files are placed in local memory). Each GOP contains several frames to ensure independent decoding capabilities. In practical applications, the GOP length is usually set to 2-4 seconds to balance the compression efficiency, fault tolerance, random access delay, and decoding complexity.

[0058] Through the above steps, the transport stream segments are generated in units of GOPs, ensuring the independent decoding capability of the transport stream segments, ensuring that there will be no problems in the subsequent playback decoding process after storage, thereby improving the user experience; and the TS file generated in units of GOPs with a length of 2-4 seconds is smaller than the cache file corresponding to the slice length of 10-30 seconds in the traditional processing method, combined with the subsequent "streaming upload + instant deletion" mechanism, thereby relieving the device running pressure caused by the local memory tension of the camera.

[0059] In step S202, a cloud server connection parameter is requested from a signaling server, and a data channel is established with the cloud server.

[0060] The audio and video collection terminal keeps communication with the signaling server through the signaling channel, and requests the dynamic connection parameter of the cloud server from the signaling server. Based on these parameters, the audio and video collection terminal establishes an independent data transmission channel with the cloud server, which is used for streaming audio and video data.

[0061] Specifically, the process is implemented as follows:

[0062] (1) Signaling channel establishment: the audio and video collection terminal establishes a long connection with the signaling server through the TCP protocol;

[0063] (2) Parameter acquisition: dynamically acquire the URL address and security authentication information stored in the cloud server from the signaling server;

[0064] (3) Data channel creation: use the acquired URL address to establish an encrypted data channel based on the HTTPS protocol with the cloud server.

[0065] Through the separation of the signaling and data channels in the above steps, the decoupling of the control instruction and the streaming data transmission is realized, avoiding the blocking of the device state reporting and other key signals by large flow video.

[0066] In step S203, the current transport stream segment is uploaded to the cloud server through the data channel, and the current transport stream segment buffered in the audio and video collection terminal is deleted after receiving the upload success response of the cloud server, and the remaining transport stream segments are continuously transmitted until a termination signal is detected; the remaining transport stream segments refer to the segments generated by encoding and encapsulating the remaining video stream; the remaining video stream is the video stream obtained from the real-time video stream except the current video stream.

[0067] Through the above data channel, the audio and video collection terminal uploads the current generated current transport stream segment to the cloud server, the cloud server receives the current transport stream segment, and generates an upload success signal after successful reception; the audio and video collection terminal detects the signal generated by the cloud server for the current transport stream segment upload success, and synchronously deletes the current transport stream video buffered in the local audio and video terminal. At the same time of uploading the current transport stream segment, the audio and video collection terminal also synchronously continues to acquire the remaining video stream, generates the corresponding transport stream segment in units of GOP, and uploads the transport stream segment to the cloud server, so as to continuously execute the above generation-upload-deletion steps until the termination information is received.

[0068] Through the above steps, the effect of "generating, uploading and deleting simultaneously" of the transport stream segment in the audio and video collection terminal is realized, and the memory pressure of the local audio and video collection terminal is reduced. At the same time, compared with the cache file of 10-30 seconds long slice generated in the traditional video processing process, the length of the transport video stream in the present scheme is shorter (2-4 seconds), so the local cache pressure is significantly reduced, and the problem that the running efficiency of the audio and video collection terminal is affected due to the limitation of local memory is effectively avoided. In addition, compared with the traditional "storage and then upload" mode, the streaming upload in the present scheme realizes "generating and uploading simultaneously", which solves the problem of upload delay.

[0069] Step S204, in response to the termination signal, a termination request is sent to the cloud server; the cloud server performs a merging operation on all transport stream segments accumulated this time based on the termination request to generate a complete transport stream file and store it.

[0070] When the audio and video collection terminal receives the signal of uploading termination, the audio and video collection terminal sends a termination request to the cloud server, terminates the upload of the transport stream segment, and ends this upload. After the cloud server receives the termination request, it performs a merging operation on all transport stream segments accumulated this time to generate a complete transport stream file and store it.

[0071] Through the above steps, the cloud server integrates multiple transport stream segments in this upload period, generates a complete transport stream file for storage, effectively reduces the storage fragmentation of the cloud server, reduces the management pressure of the cloud server, and improves the data processing efficiency of the cloud server.

[0072] Through the above steps, the audio and video acquisition terminal generates a transport stream segment in units of GOPs, and uploads the transport stream segment to the cloud server through a streaming upload mode of "generating and uploading at the same time". At the same time, based on the response of the cloud server upload success, the transport stream segment cached locally is deleted immediately, and the management mode of "immediate generation-immediate upload-immediate deletion" is realized for the transport stream segment. The memory occupation of the transport stream segment cache for the audio and video acquisition terminal memory is obviously reduced, and the effective management of the local memory of the audio and video acquisition terminal is realized. At the same time, the audio and video acquisition terminal terminates the upload of the transport stream segment according to the termination signal, and the cloud server stores the multiple transport stream segments in the current upload period after merging; in this process, the cloud server merges the scattered transport stream segments into complete transport stream files for storage, effectively reducing the storage fragmentation of the cloud server, thereby reducing the management pressure of the cloud server and relieving the index burden and access delay caused by massive small files.

[0073] In some embodiments, the termination signal includes a first termination signal; and the remaining transport stream segments are continuously transmitted until the termination signal is detected, including:

[0074] The remaining transport stream segments are continuously transmitted, and when it is detected that the audio and video acquisition terminal uploads through the current data channel for a time length accumulated to reach a first time threshold, the first termination signal is generated, and in response to the first termination signal, a first termination request is sent to the cloud server.

[0075] Wherein, the termination signal refers to the first termination signal generated by the audio and video acquisition terminal when the continuous upload time length accumulates to reach the preset first time threshold, and in response to the above first termination signal, the audio and video acquisition terminal sends a first termination request to the cloud server.

[0076] Specifically, the priority camera generates TS files in units of GOPs for embodiments, and sets the first time threshold to 60 seconds, for example, Figure 3 The above technical solutions are described in detail as follows.

[0077] (1) After the camera is started, the video is continuously recorded and TS files are generated in units of GOPs (about 2-4 seconds each);

[0078] (2) The camera maintains a TCP long connection with the signaling server, and sends a request to the signaling server to obtain an object storage URL; after the signaling server receives the above request, the object storage URL is returned to the camera;

[0079] (3) The camera establishes an HTTPS connection with the cloud server through the URL provided by the signaling server;

[0080] (4) The camera streams the first TS file to the cloud server via an HTTPS connection through HTTP POST; the cloud server feeds back a signal of successful uploading to the camera if it successfully receives the uploaded TS file;

[0081] (5) After receiving the signal of successful uploading, the camera deletes the corresponding TS file locally, while a new TS file is being generated and uploaded to the cloud server;

[0082] (6) The camera is internally provided with a timer, which starts counting from the uploading of the first TS file to count the uploading duration of the TS file;

[0083] (7) When the accumulated uploading duration reaches 60 seconds, the camera generates a termination signal and sends a HTTP request to close the instruction to the cloud server;

[0084] (8) The cloud server receives the above instruction number, integrates all the uploaded TS files in this period of time to form a complete video file and store it, completing a complete video uploading cycle.

[0085] In the above scheme, the termination signal is a first termination signal triggered by a first time threshold. When the accumulated uploading duration reaches the first time threshold (e.g., 60 seconds), the camera automatically terminates the uploading. By setting the first time threshold, the video file stored in the cloud server maintains a standard duration (e.g., 1 minute per segment), improving the storage efficiency and playback experience. In addition, it also avoids the problem of excessively long merged files caused by the unconstrained uploading of the transport stream segments, thereby affecting the retrieval efficiency of the cloud server, and improves the overall operation efficiency of the cloud server.

[0086] In some embodiments, the termination signal includes a second termination signal; the remaining transport stream segments are continuously transmitted until the termination signal is detected, including:

[0087] The remaining transport stream segments are continuously transmitted, and when a special event alarm state is detected, the second termination signal is generated, and a second termination request is sent to the cloud server in response to the second termination signal.

[0088] The special event alarm state refers to an alarm state generated by the detection of an irregular event by the audio and video acquisition terminal, such as the detection of an intrusion event or other abnormal events by the audio and video acquisition terminal, or the occurrence of a specific event (such as the appearance of a designated monitoring target), which immediately generates a corresponding alarm state.

[0089] The details of the technical solution are described below with the camera detecting an intrusion event as an example.

[0090] The camera generates a TS file in units of GOPs, and continuously performs the generation-uploading-deleting steps described above until the camera detects a termination signal, wherein the termination signal refers to a second termination signal generated when the camera detects a special event alarm state.

[0091] When the camera detects an intrusion event, the camera immediately enters a special event alarm state and generates a second termination signal. In response to the second termination signal, the camera sends a second termination request to the cloud server and stops uploading the TS file. After receiving the second termination request, the cloud server merges all TS files uploaded this time into a complete transport stream file and stores it.

[0092] In the above process, when the camera detects an intrusion event, it means that the video stream generated by the camera in the future is an abnormal video stream, which is different from the regular video stream before the intrusion alarm event occurs. At this time, the camera generates a second termination signal and sends a second termination request to the cloud server, while terminating the uploading of the TS file generated by the abnormal video stream; the cloud server integrates the TS files received before the intrusion alarm event occurs to generate a complete transport stream file and stores it. Thus, on the cloud server, the complete transport stream file generated by the cloud server for the regular video stream and the complete transport stream file generated for the abnormal video stream can be stored separately, ensuring that the special event transport stream file is stored independently and facilitating accurate retrieval afterwards.

[0093] Based on the same principle described above, when a specified monitoring target preset in the camera appears, the camera immediately enters a special event alarm state and terminates the uploading of the TS file, so that the transport stream segment generated by the video stream containing the specified monitoring target described above can be stored separately on the cloud server, facilitating subsequent accurate search and analysis.

[0094] Unlike the traditional scheme that cannot quickly segment special event videos, the above scheme realizes quick positioning of special events, avoids special event videos being submerged in lengthy files, and improves security efficiency.

[0095] In some embodiments, when the special event is detected, the method further includes, for the abnormal video stream of the special event:

[0096] Lowering the first time threshold to a target value to generate a second time threshold;

[0097] Encoding and packaging the abnormal video stream in units of GOPs to generate an abnormal transport stream segment;

[0098] requesting a new connection parameter of the cloud server from the signaling server, establishing a new data channel with the cloud server, and uploading the abnormal transport stream segments to the cloud server through the new data channel;

[0099] calling the second time threshold, and sending a third termination signal to the cloud server based on the second time threshold, wherein the cloud server is configured to perform a merging operation on all the abnormal transport stream segments accumulated this time based on the third termination signal, generate a new complete transport stream file, and store the new complete transport stream file;

[0100] resuming calling the first time threshold when the special event alarm state is detected to end.

[0101] The above camera detecting an intrusion event is taken as an example to illustrate details of the present solution.

[0102] When the camera detects an intrusion event, the camera reduces the time threshold, for example, from 60 seconds to 10 seconds, establishes a new data channel for the abnormal video stream, uploads the TS file generated by the abnormal video stream to the cloud server through the new data channel, and performs the uploading task according to the new time threshold, and the cloud server performs the merging operation according to the new time threshold and stores the corresponding complete transport stream file. When the intrusion event ends, the original time threshold is resumed.

[0103] Compared with the problem that the traditional fixed threshold cannot capture high-frequency key pictures, the present solution dynamically adjusts the time threshold, reduces the threshold during the special event, generates finer segments, avoids key frame loss, so that the complete transport stream file stored in the cloud can retain more key frames, and realizes fine recording of special events. At the same time, the original time threshold is resumed after the event ends, avoiding long-term resource waste and realizing intelligent allocation of resources.

[0104] In some embodiments, the method further comprises:

[0105] maintaining the data channel established when the current transport stream segment is first uploaded;

[0106] multiplexing the same data channel to continuously perform real-time uploading processing on the remaining transport stream segments through the data channel.

[0107] Continuing with the above example of the camera, in the present solution, the camera continuously monitors the generation of new TS files and uploads these new TS files to the cloud server using the previously established HTTPS connection, avoiding the additional overhead caused by reestablishing the connection each time.

[0108] In some of the embodiments, after the encoding and packaging of the acquired video stream in the GOP unit to generate the transport stream segment, the method further comprises:

[0109] If the number of the cached transport stream segments in the audio and video acquisition terminal is detected to exceed the maximum number threshold, the earliest generated transport stream segment is deleted in the time sequence of the generation of the cached transport stream segment.

[0110] Continuing with the above camera example, when the network quality is poor, the network is interrupted, or other special situations, it is possible to cause the slow upload / pause upload of the TS files generated by the camera, and the local cache of the camera occupies the limited local memory space of the camera. At this time, the maximum number of TS file caches can be set in the camera memory resources, and when the number of TS files cached locally by the camera exceeds the above preset value, the camera will delete the earliest generated TS file according to the generation time of the TS file.

[0111] Through the above method of automatically cleaning the earliest file when the camera cache is over-limit, the local cache overflow problem caused by network fluctuations is avoided, and the continuous operation of the device is ensured. At the same time, by preferentially retaining the latest data, information loss is reduced.

[0112] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0113] The embodiment also provides an audio and video streaming direct storage to cloud system, which is used to implement the above embodiment and preferred embodiment, and will not be described again. As used below, the terms "module", "unit", "sub-unit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiment is preferably implemented in software, hardware, or a combination of software and hardware is also possible and is contemplated.

[0114] Figure 4 is a structural block diagram of an audio and video streaming direct storage to cloud system according to the embodiment of the application, as Figure 4 shown, the system includes: an audio and video acquisition terminal 410, a cloud server 420, and a signaling server 430; the audio and video acquisition terminal 410 is connected to the signaling server 430 through a signaling channel, and acquires connection parameters of the cloud server 420; based on the above connection parameters, the audio and video acquisition terminal 410 establishes an independent data channel with the cloud server 420, and streams audio and video data.

[0115] The audio and video acquisition terminal 410 is configured to execute the method for streaming and storing the audio and video to the cloud as described above.

[0116] The cloud server 420 is further configured to perform a merging operation on all transport stream segments accumulated this time based on the termination request, generate a complete transport stream file, and store the complete transport stream file.

[0117] In the above embodiment, the audio and video acquisition terminal 410 acquires the connection parameters of the cloud server 420 through the signaling server 430, establishes an independent data channel with the cloud server 420 for streaming upload through the connection parameters, and the cloud server 420 is configured to merge the transport stream segments and store the complete transport stream file formed after the merging.

[0118] In the above scheme, the signaling server is responsible for address allocation, and the cloud server is responsible for data merging and storage. This signaling and data separation method can make the signaling server lightweight to process control instructions and not be squeezed by data stream bandwidth, support concurrent access of ten thousand devices, and also enable independent expansion of the cloud server.

[0119] In some embodiments, the cloud server is further configured to verify the time sequence continuity of the transport stream segments and check the frame header integrity of each transport stream segment.

[0120] The time sequence check ensures the receiving order of the transport stream segments, and the frame header check ensures the decodability of the transport stream segments. Through the double verification of the cloud server, the time sequence continuity and the frame header integrity of the transport stream segments are ensured, the situation that the complete transport stream file stored subsequently is damaged due to transmission errors (such as screen flashing or freezing caused by network packet loss) is avoided, the complete transport stream file after merging is 100% decodable, and playback failure is avoided.

[0121] In some embodiments, the audio and video acquisition terminal is further configured to generate index information for the complete transport stream file and report the index information to the signaling server, and the signaling server is further configured to receive the index information and store the received index information.

[0122] The audio and video acquisition terminal generates index information of the complete transport stream file and reports the index information to the signaling server for storage. The index information includes the path, duration, size, and timestamp of the complete transport stream file after merging.

[0123] The above method can quickly locate the required file in a large number of small files through centralized management of the index information by the signaling server, improves the efficiency of target video retrieval, and separates the index and storage, which is more convenient for the management of the overall system.

[0124] In some embodiments, the system further comprises a user terminal;

[0125] The user terminal is configured to send a video playback request to the signaling server;

[0126] The signaling server is further configured to parse the video playback request and obtain time range information carried by the video playback request;

[0127] The signaling server is further configured to retrieve the index information, determine target index information matching the time range information, obtain a storage path in the target index information, and return a video access address to the user terminal; wherein the video access address points to the complete transport stream file stored in the cloud server.

[0128] When the user terminal initiates a playback request, the signaling server obtains the time range information of the corresponding video through parsing of the playback request of the user terminal, performs retrieval and matching according to the time range information, determines the target index information stored in the signaling server, and returns the access address of the complete transport stream file corresponding to the target index information to the user terminal. The user terminal accesses the complete transport stream file and decodes it to obtain the corresponding playback video stream, so as to realize the playback requirement.

[0129] It should be noted that each of the above modules can be a functional module or a program module, which can be implemented by software or hardware. For the modules implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can also be located in different processors in any combination. In this embodiment, specific examples can be referred to the examples described in the above embodiments and optional implementation manners, which will not be described herein.

[0130] The embodiment also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned audio and video stream storage-to-cloud method embodiments.

[0131] Optionally, the electronic device can further comprise a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0132] Optionally, in this embodiment, the processor can be configured to execute the following steps through the computer program:

[0133] S1, encoding and packaging the obtained current video stream in units of GOPs to generate a current transport stream segment;

[0134] S2, requesting a cloud server connection parameter from a signaling server, and establishing a data channel with the cloud server;

[0135] S3, uploading the current transport stream segment to the cloud server through the data channel, synchronously deleting the current transport stream segment buffered in the audio and video collection terminal after receiving a successful uploading response from the cloud server, continuously uploading remaining transport stream segments until a termination signal is detected; the remaining transport stream segments refer to transport stream segments generated by encoding and encapsulating remaining video streams in units of GOPs; the remaining video streams refer to video stream data other than the current video stream in the acquired real-time video stream;

[0136] S4, in response to the termination signal, sending a termination request to the cloud server; based on the termination request, the cloud server performs a merging operation on all transport stream segments accumulated this time to generate a complete transport stream file and store it.

[0137] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation manners, and this embodiment will not be repeated here.

[0138] In addition, in combination with the audio and video stream direct storage to cloud method in the above embodiments, the present embodiment can provide a storage medium for implementation. The storage medium has a computer program stored thereon; the computer program is executed by a processor to implement any one of the audio and video stream direct storage to cloud methods in the above embodiments.

[0139] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0141] Those skilled in the art should understand that any combination of the technical features of the above-mentioned embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above-mentioned embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0142] The above-mentioned embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of protection of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A method for directly storing audio and video streams in the cloud, executed by an audio and video acquisition terminal, characterized in that: The method comprises the following steps: encoding and packaging the obtained current video stream in GOP units to generate a current transport stream segment; requesting a cloud server connection parameter from a signaling server and establishing a data channel with the cloud server; uploading the current transport stream segment to the cloud server through the data channel, deleting the current transport stream segment buffered in the audio and video collection terminal after receiving a successful upload response from the cloud server, and continuously transmitting the remaining transport stream segments until a termination signal is detected; the remaining transport stream segments refer to the segments generated by encoding and packaging the remaining video stream; the remaining video stream is the video stream obtained from the real-time video stream except the current video stream; sending a termination request to the cloud server in response to the termination signal; The cloud server performs a merging operation on all transport stream segments uploaded this time based on the termination request to generate and store a complete transport stream file.

2. The method of claim 1, wherein, The termination signal includes a first termination signal; the continuous transmission of the remaining transport stream segments until the termination signal is detected includes: continuously transmitting the remaining transport stream segments, and when it is detected that the audio and video collection terminal uploads through the data channel for a time length accumulated to a preset first time threshold, the first termination signal is generated, and a first termination request is sent to the cloud server in response to the first termination signal. 3.The method of claim 1, wherein, The termination signal includes a second termination signal; the continuous transmission of the remaining transport stream segments until the termination signal is detected includes: continuously transmitting the remaining transport stream segments, and when a special event alarm state is detected, the second termination signal is generated, and a second termination request is sent to the cloud server in response to the second termination signal.

4. The method of claim 3, wherein, When the special event is detected, the method further comprises: downgrade the first time threshold to a target value to generate a second time threshold; encoding and packaging the abnormal video stream in GOP units to generate an abnormal transport stream segment; requesting new connection parameters of the cloud server from the signaling server and establishing a new data channel with the cloud server; uploading the abnormal transport stream segment to the cloud server through the new data channel; calling the second time threshold; sending a third termination signal to the cloud server based on the second time threshold; wherein the cloud server is configured to perform a merging operation on all abnormal transport stream segments uploaded this time based on the third termination signal to generate and store a new complete transport stream file; when the special event alarm state ends, the first time threshold is restored.

5. The method of claim 1, wherein, The method further comprises: maintaining the data channel established when the current transport stream segment is first uploaded; multiplexing the same data channel to continuously upload the remaining transport stream segments in real time through the data channel.

6. The method of streaming audio and video to the cloud according to any one of claims 1-5, wherein, After the step of encoding and packaging the obtained video stream in GOP units to generate a transport stream segment, the method further comprises: If the number of the cached transport stream segments in the audio and video collection terminal is detected to exceed the maximum threshold, the earliest generated transport stream segment is deleted according to the generation time sequence of the cached transport stream segments.

7. A system for streaming audio and video directly to the cloud, the system comprising: The system comprises an audio and video collection terminal, a cloud server and a signaling server. The signaling server is configured to establish a data channel between the audio and video collection terminal and the cloud server. The audio and video collection terminal is configured to perform the method for directly storing the audio and video stream to the cloud according to any one of claims 1 to 6. The cloud server is further configured to perform a merging operation on all the transport stream segments accumulated this time based on the termination request, generate a complete transport stream file and store the complete transport stream file.

8. The system of claim 7, wherein, The cloud server is further configured to verify the time sequence continuity of the transport stream segments and check the frame header integrity of each transport stream segment.

9. The system of claim 7, wherein, The audio and video collection terminal is further configured to generate index information for the complete transport stream file and report the index information to the signaling server.

10. The system of claim 9, wherein, The system further comprises a user terminal. The user terminal is configured to send a video playback request to the signaling server. The signaling server is further configured to analyze the video playback request and obtain time range information carried by the video playback request. The signaling server is further configured to retrieve the index information, determine target index information matched with the time range information, obtain a storage path in the target index information, and return the storage path to the user terminal; wherein the storage path points to the complete transport stream file stored in the cloud server.

Images