A method and system for obtaining a video thumbnail of a cloud server by using a client

Abstract

The present invention provides a method and system for obtaining a cloud server video thumbnail using a client, aiming to solve the technical problem that the cloud server has a significantly increased load due to decoding and image format conversion. The method for obtaining a cloud server video thumbnail using a client includes: the cloud server stores audio and video data in segments on the cloud server; the event is stored as an event list on the cloud server with a unique camera identifier, the event start UTC time, and the event end UTC time; the client pulls the event list from the cloud server; the client has a corresponding thumbnail cache; the thumbnail in the client is directly read; if not, the first IDR frame of the audio and video corresponding to each event in the event list is asynchronously pulled from the cloud server using the camera unique identifier, the event start UTC time, and the event end UTC time as an index; the client decodes the pulled IDR frame, performs image format conversion, and obtains a thumbnail. The present invention reduces the load on the cloud server.

Description

Technical Field

[0001] The present invention belongs to the technical field of camera audio and video processing, and in particular relates to a method and system for obtaining a video thumbnail of a cloud server by using a client. Background Art

[0002] At present, network video surveillance cameras are widely used. The audio and video obtained by network video surveillance cameras are generally in H.264 and H.265 formats. In addition to live broadcast scenes, the audio and video generated by surveillance cameras are basically stored in the cloud. Users can watch them back through the client. The watched videos will also generate a large number of events based on the detection of people, pets, abnormal sounds, etc. in the video.

[0003] When a user uses a client to preview a list of events within a period of time, it is generally necessary to display a thumbnail of the event at the same time. Currently, it is more common that when a user previews an event list, the H.264 / H.265IDR frame data of the event video is first obtained from the cloud server, and then the IDR frame is decoded, and then the image format is converted and returned to the user's PC web client or mobile client for display. If the user previews a large number of events in the event list and a large number of users are previewing at the same time, the cloud server will be significantly loaded with decoding and image format conversion, resulting in resource shortages and increased costs.

[0004] In some existing technologies, in order to avoid repeated generation of thumbnails for the same event when previewing on different clients, the generated event thumbnails are cached. When other clients preview, the thumbnails cached by the cloud server are directly returned. Although this reduces the pressure of repeated decoding and image conversion on the cloud server, it increases the storage cost of the server. Summary of the invention

[0005] The present invention provides a method and system for obtaining a cloud server video thumbnail by using a client, aiming to solve the technical problem in the prior art that event thumbnails are generated by a cloud server, and the cloud server load is significantly increased due to decoding and image format conversion.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention provides a method for obtaining a video thumbnail of a cloud server by using a client, comprising the following steps:

[0008] S1. The cloud server receives the audio and video data uploaded by the camera, and stores the audio and video data in segments according to the time length on the cloud server; when receiving an event acquired by the camera, the event is stored as an event list on the cloud server with the camera's unique identifier, the event start UTC time, and the event end UTC time;

[0009] S2, the client pulls the event list from the cloud server according to the specified camera and time range; determines whether the client has a corresponding thumbnail cache; if so, directly reads the thumbnail in the client; if not, executes step S3;

[0010] S3, using the camera unique identifier, the event start UTC time, and the event end UTC time as indexes, asynchronously pulls the first IDR frame of the audio and video corresponding to each event in the event list from the audio and video shard storage on the cloud server;

[0011] S4, the client decodes the pulled IDR frame and converts the image format to obtain a thumbnail;

[0012] S5. Display the thumbnail in the corresponding event list, and cache it to the client using the camera unique identifier, the event start UTC time, and the event end UTC time as indexes.

[0013] A further improved solution: the video adopts the H.264 format or the H.265 format.

[0014] Further improved solution: the client includes a mobile phone or a computer.

[0015] Further improved solution: In step S4, the client decodes the pulled IDR frame and converts the image format to obtain a thumbnail; specifically, the client decodes the pulled IDR frame into a texture, scales the texture, and finally converts the texture into a jpeg thumbnail.

[0016] A further improved solution: when the client is a mobile phone, the IDR frame pulled by the client is decoded into a texture using the client's hardware decoder, and the texture is scaled, and finally the texture is converted into a jpeg thumbnail.

[0017] Further improved solution: When the mobile phone is iOS system, the hardware decoder uses VideoToolBox; when the mobile phone is Android system, the hardware decoder uses MediaCodec.

[0018] A further improved solution: when the client is a computer web page, the IDR frame pulled by the client is decoded and scaled using the WebAssembly component and finally converted into a jpeg thumbnail.

[0019] Further improved solution: When the camera obtains an event, the camera obtains the generated event through AI detection; the UTC time of the IDR frame closest to the time point of the event generation is used as the UTC time of the event start; the end time of the event detection is used as the UTC time of the event end, forming a complete event.

[0020] A further improved solution: the events include humanoid events, pet events and unusual noise events.

[0021] In a second aspect, the present invention provides a system for obtaining a video thumbnail of a cloud server by using a client, including a client, a server, and a camera;

[0022] The cloud server receives the audio and video data uploaded by the camera, and stores the audio and video data in slices according to time length in the cloud server; when receiving an event acquired by the camera, the event is stored as an event list in the cloud server with the camera unique identifier, the event start UTC time and the event end UTC time;

[0023] The client pulls the event list from the cloud server according to the specified camera and time range; determines whether there is a corresponding thumbnail cache on the client;

[0024] If it exists, the thumbnail in the client is read directly;

[0025] If it does not exist, the first IDR frame of the audio and video corresponding to each event in the event list is asynchronously pulled from the audio and video segment storage of the cloud server using the camera's unique ID, the event start UTC time, and the event end UTC time as indexes; the client decodes the pulled IDR frame and converts the image format to obtain a thumbnail; the thumbnail is displayed in the corresponding event list and cached to the client using the camera's unique ID, the event start UTC time, and the event end UTC time as indexes.

[0026] A further improved solution: the video adopts the H.264 format or the H.265 format.

[0027] Further improved solution: the client includes a mobile phone or a computer.

[0028] A further improved solution: the client decodes the pulled IDR frame and converts the image format to obtain a thumbnail; specifically, the client decodes the pulled IDR frame into a texture, scales the texture, and finally converts the texture into a jpeg thumbnail.

[0029] A further improved solution: when the client is a mobile phone, the IDR frame pulled by the client is decoded into a texture using the client's hardware decoder, and the texture is scaled, and finally the texture is converted into a jpeg thumbnail.

[0030] Further improved solution: When the mobile phone is iOS system, the hardware decoder uses VideoToolBox; when the mobile phone is Android system, the hardware decoder uses MediaCodec.

[0031] A further improved solution: when the client is a computer web page, the IDR frame pulled by the client is decoded and scaled using the WebAssembly component and finally converted into a jpeg thumbnail.

[0032] Further improved solution: When the camera obtains an event, the camera obtains the generated event through AI detection; the UTC time of the IDR frame closest to the time point of the event generation is used as the UTC time of the event start; the end time of the event detection is used as the UTC time of the event end, forming a complete event.

[0033] A further improved solution: the events include humanoid events, pet events and unusual noise events.

[0034] The beneficial effects of the present invention are:

[0035] In the present invention, the role of the cloud server is to store the audio and video data in slices according to the time length in the cloud server, and store the event as an event list in the cloud server with the unique camera identifier, the event start UTC time and the event end UTC time; the cloud server is mainly used to store audio and video data and time lists. The role of the client is to asynchronously pull the first IDR frame of the audio and video corresponding to each event in the event list from the audio and video slice storage of the cloud server with the unique camera identifier, the event start UTC time and the event end UTC time as indexes; the client decodes the pulled IDR frame, and performs image format conversion to obtain a thumbnail.

[0036] It can be seen from this that in the present invention, thumbnails of events are generated through the cooperation of the client and the cloud server; the client mainly plays the role of storing audio and video data and event lists, and the client mainly plays the role of generating thumbnails. The division of labor and cooperation between the two will only transfer the generation of thumbnails to the client, solving the problem of a significant increase in the load of the cloud server due to decoding and image format conversion; in addition, the storage of audio and video data and event lists is still retained on the cloud server, which can enable different clients to still access them normally without increasing the storage burden of the client.

[0037] In addition, the client has a cache function for the generated thumbnails. The next time you access events in the same time period of the same camera, there is no need to generate them again, thus avoiding the repeated generation of thumbnails. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flow chart of the method for obtaining a cloud server video thumbnail by using a client in the present invention.

[0040] Figure 2 It is a schematic diagram of the logical framework of the system for obtaining the video thumbnail of the cloud server by using the client in the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present invention.

[0042] Embodiment 1:

[0043] like Figure 1 As shown, this embodiment provides a method for obtaining a cloud server video thumbnail by using a client, including the following steps:

[0044] S1. The cloud server receives the audio and video data uploaded by the camera, and stores the audio and video data in slices according to the time length on the cloud server (the audio and video data files within a period of time are regarded as one slice, such as one GOP (Group of Picture) picture group as one slice, which is convenient for retrieval); when receiving an event obtained by the camera, the event is stored as an event list on the cloud server with the camera's unique identifier, the event start UTC time (Coordinated Universal Time, also known as World Unified Time) and the event end UTC time.

[0045] Specifically, the video adopts a format such as H.264 format or H.265 format.

[0046] Among them, the unique identification of the camera refers to the independent identity established for each camera, which can use an independent number as an identification.

[0047] Among them, a large number of events will be generated in the audio and video, including human events, pet events, and abnormal sound events. The video of each event will contain multiple IDR frames; when the camera obtains an event, the camera obtains the generated event through AI detection; the UTC time of the generation of the IDR frame closest to the generation time point of the event is used as the start UTC time of the event; when the event detection end time is used as the end UTC time of the event, a complete event is formed; several events in the video are stored as an event list on the cloud server with the unique camera identifier, the event start UTC time and the event end UTC time.

[0048] Among them, IDR (Instantaneous Decoding Refresh) frames belong to I frames (Iframe), also known as internal pictures, but I frames are not necessarily IDR frames; when the decoder receives an IDR frame, it clears the reference cache, but does not clear the reference cache when receiving an I frame, that is to say, for the frames after the IDR frame, the decoder will not refer to any frames before the IDR frame for decoding, and for the frames after a certain I frame, the decoder may refer to the frames before the I frame for decoding.

[0049] S2. The client pulls the event list from the cloud server according to the specified camera and time range; determines whether the client has a corresponding thumbnail cache; if so, directly reads the thumbnail in the client; if not, executes step S3.

[0050] Specifically, the client includes but is not limited to a mobile phone or a computer (computer web client);

[0051] The thumbnail refers to a small image obtained after the image is processed by scaling, conversion, etc.

[0052] There are generally multiple events within the camera and time range specified by the client, and correspond to a list of events within a time period.

[0053] In addition, by determining whether the client has a corresponding thumbnail cache, repeated generation of thumbnails can be avoided, saving the client's computing power.

[0054] S3. Using the camera's unique ID, the event start UTC time, and the event end UTC time as indexes, asynchronously pull the first IDR frame of the audio and video corresponding to each event in the event list from the audio and video segment storage on the cloud server.

[0055] The video of each event contains multiple IDR frames, and the first IDR frame is processed into a thumbnail.

[0056] S4. The client decodes the pulled IDR frame and converts the image format to obtain a thumbnail.

[0057] In step S4, the client decodes the pulled IDR frame and converts the image format to obtain a thumbnail; specifically, the client decodes the pulled IDR frame into a texture, scales the texture, and finally converts the texture into a jpeg thumbnail;

[0058] When the client is a mobile phone, the IDR frame pulled by the client is decoded into a texture using the client's hardware decoder, and the texture is scaled, and finally the texture is converted into a jpeg thumbnail. Specifically, when the mobile phone is an iOS system (a mobile operating system developed by Apple), the hardware decoder uses VideoToolbox (a video hardware encoder and decoder provided by Apple's operating system); when the mobile phone is an Android system (a free and open source operating system based on the Linux kernel), the hardware decoder uses MediaCodec (a media codec provided by the Android system).

[0059] When the client is a computer web page, the IDR frame pulled by the client is decoded and scaled using the WebAssembly (encapsulating FFmpeg h.264 / h.265 decoding and jpeg compression part) component, and finally converted into a jpeg thumbnail.

[0060] S5. Display the thumbnail in the corresponding event list, and cache it to the client using the camera unique identifier, the event start UTC time, and the event end UTC time as indexes.

[0061] Embodiment 2:

[0062] See also Figure 2 ,This embodiment provides a system for obtaining a video thumbnail of a cloud server by using a client, including a client, a server and a camera;

[0063] The cloud server receives the audio and video data uploaded by the camera, and stores the audio and video data in slices according to time length in the cloud server; when receiving an event acquired by the camera, the event is stored as an event list in the cloud server with the camera unique identifier, the event start UTC time and the event end UTC time;

[0064] The client pulls the event list from the cloud server according to the specified camera and time range; determines whether there is a corresponding thumbnail cache on the client;

[0065] If it exists, the thumbnail in the client is read directly;

[0066] If it does not exist, the first IDR frame of the audio and video corresponding to each event in the event list is asynchronously pulled from the audio and video segment storage of the cloud server using the camera's unique ID, the event start UTC time, and the event end UTC time as indexes; the client decodes the pulled IDR frame and converts the image format to obtain a thumbnail; the thumbnail is displayed in the corresponding event list and cached to the client using the camera's unique ID, the event start UTC time, and the event end UTC time as indexes.

[0067] The video is in H.264 format or H.265 format. The events include humanoid events, pet events and abnormal sound events.

[0068] The client includes a mobile phone or a computer. The client decodes the pulled IDR frame and converts the image format to obtain a thumbnail; specifically, the client decodes the pulled IDR frame into a texture, scales the texture, and finally converts the texture into a jpeg thumbnail.

[0069] When the client is a mobile phone, the IDR frame pulled by the client is decoded into a texture using a hardware decoder of the client, and the texture is scaled, and finally the texture is converted into a jpeg thumbnail.

[0070] When the mobile phone is running on iOS, the hardware decoder uses VideoToolBox; when the mobile phone is running on Android, the hardware decoder uses MediaCodec.

[0071] When the client is a computer web page, the IDR frame pulled by the client is decoded and scaled using the WebAssembly component and finally converted into a jpeg thumbnail.

[0072] Among them, when the camera obtains an event, the camera obtains the generated event through AI detection; the UTC time of the IDR frame closest to the time point of the event generation is used as the UTC time of the event start; when the event detection ends, it is used as the UTC time of the event end, forming a complete event.

[0073] The present invention is not limited to the above-mentioned optional implementation modes. Anyone can derive other various forms of products under the inspiration of the present invention. However, no matter what changes are made in the shape or structure, all technical solutions that fall within the scope defined by the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A method for obtaining a video thumbnail of a cloud server by using a client, characterized in that: The following steps are involved: S1, the cloud server receives the audio and video data uploaded by the camera, and stores the audio and video data in segments according to the time length on the cloud server; When an event acquired by the camera is received, the event is stored as an event list on the cloud server with the camera's unique ID, the event start UTC time, and the event end UTC time; S2. The client pulls the event list from the cloud server according to the specified camera and time range; determines whether there is a corresponding thumbnail cache on the client; If it exists, the thumbnail in the client is read directly; If not, execute step S3; S3, using the camera unique identifier, the event start UTC time, and the event end UTC time as indexes, asynchronously pulls the first IDR frame of the audio and video corresponding to each event in the event list from the audio and video segment storage on the cloud server; S4, the client decodes the pulled IDR frame and converts the image format to obtain a thumbnail; S5. Display the thumbnail in the corresponding event list, and cache it to the client using the camera unique identifier, the event start UTC time, and the event end UTC time as indexes.

2. A method for obtaining a cloud server video thumbnail using a client according to claim 1, characterized in that: The video is in H.264 format or H.265 format.

3. A method for obtaining a cloud server video thumbnail using a client according to claim 1 or 2, characterized in that: The client includes a mobile phone or a computer.

4. A method for obtaining a cloud server video thumbnail by using a client according to claim 3, characterized in that: In step S4, the client decodes the pulled IDR frame and converts the image format to obtain a thumbnail; specifically, the client decodes the pulled IDR frame into a texture, scales the texture, and finally converts the texture into a jpeg thumbnail.

5. A method for obtaining a cloud server video thumbnail using a client according to claim 4, characterized in that: When the client is a mobile phone, the IDR frame pulled by the client is decoded into a texture using a hardware decoder of the client, and the texture is scaled, and finally the texture is converted into a jpeg thumbnail.

6. A method for obtaining a cloud server video thumbnail using a client according to claim 5, characterized in that: When the mobile phone is running on iOS, the hardware decoder uses VideoToolBox; when the mobile phone is running on Android, the hardware decoder uses MediaCodec.

7. A method for obtaining a cloud server video thumbnail by using a client according to claim 4, characterized in that: When the client is a computer web page, the IDR frame pulled by the client is decoded and scaled using the WebAssembly component and finally converted into a jpeg thumbnail.

8. The method for obtaining a cloud server video thumbnail by using a client according to claim 1, characterized in that: When the camera obtains an event, it uses AI detection to obtain the generated event. The UTC time of the IDR frame closest to the time point of the event is used as the start UTC time of the event. The end time of the event detection is used as the end UTC time of the event to form a complete event.

9. A method for obtaining a cloud server video thumbnail by using a client according to claim 1 or 8, characterized in that: The events include humanoid events, pet events and unusual noise events.

10. A system for obtaining a video thumbnail of a cloud server by using a client, characterized in that: Including client, cloud server and camera; The cloud server receives the audio and video data uploaded by the camera, and stores the audio and video data in slices according to time length on the cloud server; When an event acquired by the camera is received, the event is stored as an event list on the cloud server with the camera's unique ID, the event start UTC time, and the event end UTC time; The client pulls the event list from the cloud server according to the specified camera and time range; Determine whether the client has a corresponding thumbnail cache; If it exists, the thumbnail in the client is read directly; If it does not exist, the camera unique identifier, the event start UTC time, and the event end UTC time are used as indexes to asynchronously pull the first IDR frame of the audio and video corresponding to each event in the event list from the audio and video segment storage on the cloud server. The client decodes the pulled IDR frame and converts the image format to obtain a thumbnail. The thumbnail is displayed in the corresponding event list and cached to the client with the camera unique identifier, event start UTC time and event end UTC time as indexes.

Images