A multimedia data encoding method and device

Abstract

A multimedia data encoding method and device can encode different data segments in the multimedia data captured by a camera in different ways. Specifically, original data segments of different value levels can be encoded using encoding methods with different compression rates. The storage space occupied by the encoded multimedia data obtained with different compression rates is also different, thereby effectively alleviating the pressure of storing multimedia data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of information technology, and in particular to a multimedia data encoding method and device. Background Art

[0002] The video surveillance system is generally composed of a camera at the front end, a transmission system, and a storage system at the back end. The camera is mainly responsible for the acquisition of raw multimedia data. The transmission system is responsible for transmitting the encoded multimedia data acquired and encoded by the camera to the storage system. The storage system is responsible for displaying the received encoded multimedia data in real time on the display device or storing it in the storage medium. The storage system can also perform big data analysis or artificial intelligence (AI) analysis on the encoded multimedia data to obtain valuable information. Among them, the raw multimedia data can be image frames or video segments.

[0003] At present, the workflow of the video surveillance system is: the camera transmits the raw multimedia data captured in real time to the storage system for storage and analysis. Its essence is that the camera captures the raw multimedia data, then encodes the raw multimedia data, and sends it to the storage system through the transmission system for persistent storage. However, in actual scenarios, the image frames or video segments captured by the camera have a small proportion of valuable content, and a large part of the image frames or video segments consume a lot of transmission bandwidth, storage costs, and intelligent analysis costs. In the existing technology, in order to save the return bandwidth, the feature values ​​of the image frames or video segments are only returned in real time during busy hours, and the transmitted image frames or video segments on the scene are temporarily stored locally in the camera, and the image frames or video segments are only asynchronously returned during idle hours. In this technology, although the pressure of transmitting image frames or video segments during busy hours is alleviated, the amount of data cannot be reduced, and the transmitted image frames or video segments cannot be transmitted together with the feature values, which will cause incompleteness of the supporting data, thereby affecting the real-time decision-making of the back-end. Summary of the invention

[0004] The embodiments of the present application provide a multimedia data encoding method and device, which can identify the value level of the original data segments in the original multimedia data, so that the original data segments of different value levels can be encoded using encoding methods with different compression rates. In this way, when the encoded multimedia data is transmitted back, the encoded multimedia data obtained with different compression rates occupy different transmission bandwidths and different storage spaces, which can effectively alleviate bandwidth pressure and storage pressure.

[0005] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0006] In a first aspect, a multimedia data encoding method is provided, comprising: shooting with a camera to obtain original multimedia data; determining the value level of each original data segment in the original multimedia data according to a value level rule, wherein the original multimedia data includes multiple original data segments, wherein the multiple original data segments include a first original data segment and a second original data segment, and the value level of the first original data segment is higher than the value level of the second original data segment; selecting an encoding method corresponding to the value level of each original data segment in the multiple original data segments, wherein different encoding methods correspond to different compression rates, and the compression rate of the encoding method corresponding to the first original data segment is lower than the compression rate of the encoding method corresponding to the second original data segment; encoding each original data segment in the original multimedia data according to the selected encoding method to generate encoded multimedia data corresponding to the original multimedia data.

[0007] Therefore, in the embodiment of the present application, by analyzing the value levels of different original data segments in the original multimedia data captured by the camera, the encoding methods corresponding to the value levels can be determined according to the value levels of the different original data segments, and different encoding methods correspond to different compression rates. If the compression rate of the original data segment with a high value level is low, and the compression rate of the original data segment with a low value level is high, when the encoded multimedia data corresponding to the original data segment is obtained by encoding and sent to the media management device, the storage space occupied by the encoded multimedia data with a high compression rate is smaller than the storage space occupied by the encoded multimedia data with a low compression rate. Compared with the prior art, when the same encoding method with a low compression rate is used for encoding, the storage space occupied by the encoded multimedia data obtained after compression is large. The present application can distinguish the encoding methods of different original multimedia data according to the value level, not only using one encoding method, if the original multimedia data is compressed by the encoding method with a high compression rate, the storage space occupied by the obtained encoded multimedia data is relatively small, which can save storage space.

[0008] Moreover, compared with the prior art that uses a coding method to compress data, when the compression rate is low, the size of the encoded multimedia data is large, and the bandwidth pressure during transmission is large. In the present application, the size of the encoded multimedia data obtained from the original data with a high compression rate is smaller, and the bandwidth pressure can also be alleviated when transmitting the encoded multimedia data.

[0009] In one possible design, the original data segment includes one or more of the following: a video segment, a complete image frame, or a portion of an image frame.

[0010] When the original data segment is a video segment, that is, different video segments may correspond to different encoding methods, that is, different compression rates are used. Video segments with high value levels correspond to low compression rates and high fidelity, while video segments with low value levels correspond to high compression rates and low fidelity. Similarly, when the original data segment is a complete image frame, complete image frames with low value levels correspond to high compression rates and low fidelity, while complete image frames with high value levels correspond to low compression rates and high fidelity. This allows users to visually feel clearer video segments and complete image frames with high value levels. For example, when used in security scenarios, people who need attention can be identified in a timely manner.

[0011] When the original data segment is part of an image frame, one part of the same image frame has a high value level and another part has a low value level. The same image frame can be encoded using different encoding methods, so that the original data segment with a high value level in the same image frame has a low compression rate and high fidelity when encoded, and the original data segment with a low value level has a high compression rate and low fidelity when encoded. For example, in a security scene, if multiple pedestrians are captured in the same image frame, the camera can only encode the portrait area of ​​the pedestrian that matches the value event in the value event library using a low compression rate encoding method, and encode other pedestrians using a high compression rate encoding method.

[0012] In a possible design, the value level of each original data segment in the original multimedia data is determined according to a value level rule, specifically including: identifying value events in the original multimedia data through semantic analysis, and determining the value level of the original data segment where the value event is located according to the value event, wherein the value level rule is used to describe the correspondence between the value event and the value level.

[0013] There are many ways of semantic analysis, such as computer vision analysis, video semantic analysis or AI algorithms such as machine learning. There may be objects that need attention in the image frames corresponding to the original multimedia data. The characteristics of the objects that need attention can be identified through semantic analysis, and the characteristics are compared with the characteristics of the value events in the value event library. If the comparison is successful, the value level of the object that needs attention can be determined according to the value level rules. The value level of the object that needs attention is the value level of the original data segment where the object is located. In this way, the corresponding encoding method can be used for encoding according to the determined value level. Different encoding methods have different corresponding compression rates to further save storage space of multimedia management equipment.

[0014] In a possible design, valuable events are recorded in the first original data segment; and valuable events are not recorded in the second original data segment.

[0015] That is, when the first original data segment records a valuable event and the second original data segment does not record a valuable event, the value level of the first original data segment is higher than the value level of the second original data segment. In this way, the first original data segment with a high value level can be encoded in a low compression rate, so that the visual fidelity of the first original data segment with a high value level is high.

[0016] In one possible design, the method further includes: immediately sending the first original data segment to the multimedia management device; storing the second original data segment in a local non-volatile storage medium of the camera, and then sending the second original data segment in the non-volatile storage medium to the multimedia management device.

[0017] Since the value level of the first original data segment is higher than that of the second original data segment, the first original data segment records valuable events, while the second original data segment does not record valuable events. Therefore, the first original data segment can be immediately sent to the multimedia management device through the transmission system, and the second original data segment can be temporarily stored locally in the camera. The multimedia management device includes a storage device, a display device, and a control and analysis device, so that users on the multimedia management device side can be informed of valuable events in a timely manner. The second original data segment that does not record valuable events is stored in the local non-volatile storage medium of the camera, and the second original data segment in the non-volatile storage medium is sent to the multimedia management device at idle time. On the basis of timely knowing valuable events, the storage pressure on the multimedia management device side and the bandwidth pressure of data transmission can be further alleviated. Among them, the second original data segment can also be stored locally in the camera for a storage period, and when the storage period expires, the second original data segment can be deleted.

[0018] In one possible design, the method further includes: sending the first original data segment to the multimedia management device using a high-speed network; and sending the second original data segment to the multimedia management device using a low-speed network.

[0019] Since the value level of the first original data segment is higher than that of the second original data segment, for example, when a valuable event is recorded in the first original data segment and no valuable event is recorded in the second original data segment, the encoded multimedia data obtained after encoding the first original data segment can be sent to the multimedia management device through a high-speed network so that users on the multimedia management device side can be informed of the valuable event in a timely manner. For example, in a security scenario, the first original data segment records the portrait features of a criminal suspect, so the user can quickly learn about the movements of the criminal suspect. In addition, when sending, the encoded multimedia data can also include the location information of the valuable event.

[0020] The second original data segment can be sent to the multimedia management device using a low-speed network when the camera is idle. In this way, when sending the first original data segment with valuable events, the second original data segment without recording valuable events does not occupy the bandwidth of the high-speed network, alleviating the bandwidth pressure when sending valuable events. The high-speed network can be, for example, 5G wireless transmission, and the low-speed network can be, for example, eMBB large-capacity demand network slice transmission.

[0021] In one possible design, the format of the original multimedia data includes: one of the RAW format, RGB format or YUV format, HSV format, Lab format, CMY format or YCbCr format; the format of the encoded multimedia data includes: one of the image media format or video media format.

[0022] Among them, the image frame directly generated by the sensor in the camera can be in RAW format. Depending on the design of the sensor, the format of RAW is also different. For example, the RAW format can be Bayer RGGB, RYYB, RCCC, RCCB, RGBW, CMYW and other formats. In some embodiments, the camera can also use ISP to convert RAW images of various formats into RGB format. ISP can also convert the RAW format into the first of the following formats: YUV format, HSV format, Lab format, CMY format, YCbCr format. The above formats, such as RGGB, YUV, HSV format, Lab format, CMY format, YCbCr format, can be called raw media data formats.

[0023] If the original image is encoded by the encoder in the camera, it can be converted into a media data format that is easy for the human eye to recognize and occupies less storage space. The media data formats of images can include, for example, jpeg format, bmp format, tga format, png format, and gif format. The media formats of videos can include, for example, MPEG format, AVI format, nAVI format, ASF format, MOV format, WMV format, 3GP format, RM format, RMVB format, FLV / F4V format, H.264 format, and H.265 format.

[0024] In a possible design, the camera can receive configuration information sent by the multimedia management device, and the configuration information can include value events and value level rules. The value level rules are used to describe the corresponding relationship between value events and value levels.

[0025] The configuration information may also include storage methods corresponding to the value levels, and may also include transmission methods corresponding to the value levels.

[0026] For example, when the value level of the first original data segment is higher than that of the second original data segment, the first original data segment may be immediately sent and stored on the multimedia management device side, and the second original data segment may be temporarily stored locally on the camera. The first original data segment may be transmitted to the multimedia management device using a high-speed network, and the second original data segment may be transmitted to the multimedia management device using a low-speed network. This can alleviate the storage pressure on the multimedia management device side and save storage space. It can also alleviate the bandwidth pressure when transmitting data.

[0027] In a second aspect, an electronic device is provided, including: a shooting module, used for shooting and acquiring original multimedia data;

[0028] The processing module is further used to determine the value level of each original data segment in the original multimedia data according to the value level rule, the original multimedia data includes multiple original data segments, wherein the multiple original data segments include a first original data segment and a second original data segment, and the value level of the first original data segment is higher than the value level of the second original data segment; the processing module is further used to select an encoding method corresponding to the value level of each original data segment in the multiple original data segments, wherein different encoding methods correspond to different compression rates, and the compression rate of the encoding method corresponding to the first original data segment is lower than the compression rate of the encoding method corresponding to the second original data segment; the encoding module is further used to encode each original data segment in the original multimedia data according to the selected encoding method to generate encoded multimedia data corresponding to the original multimedia data.

[0029] In one possible design, the original data segment includes one or more of the following: a video segment, a complete image frame, or a portion of an image frame.

[0030] In one possible design, the processing module is used to: identify value events in original multimedia data through semantic analysis, and determine the value level of the original data segment where the value event is located according to the value event, wherein the value level rule is used to describe the correspondence between the value event and the value level.

[0031] In a possible design, valuable events are recorded in the first original data segment; and valuable events are not recorded in the second original data segment.

[0032] In one possible design, the electronic device also includes a transceiver module, which is used to: immediately send the first original data segment to the multimedia management device; the electronic device also includes a storage module, which is used to store the second original data segment in the local non-volatile storage medium of the camera, and then send the second original data segment in the non-volatile storage medium to the multimedia management device.

[0033] In a possible design, it also includes a transceiver module, which is used to: send the first original data segment to the multimedia management device using a high-speed network; and send the second original data segment to the multimedia management device using a low-speed network.

[0034] In one possible design, the format of the original multimedia data includes: one of the RAW format, RGB format or YUV format, HSV format, Lab format, CMY format or YCbCr format; the format of the encoded multimedia data includes: one of the image media format or video media format.

[0035] According to a third aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the first aspect and any possible design of the first aspect.

[0036] According to a fourth aspect, a computer program product is provided. When the computer program product is run on a computer, the electronic device executes the method described in the first aspect and any possible design of the first aspect.

[0037] In a fifth aspect, a camera is provided, characterized in that it includes: a lens assembly for receiving light; a sensor for converting the light received by the lens assembly into original multimedia data; a processor for: determining the value level of each original data segment in the original multimedia data according to a value level rule, the original multimedia data including multiple original data segments, wherein the multiple original data segments include a first original data segment and a second original data segment, and the value level of the first original data segment is higher than the value level of the second original data segment; selecting an encoding method corresponding to the value level of each original data segment in the multiple original data segments, wherein different encoding methods correspond to different compression rates, and the compression rate of the encoding method corresponding to the first original data segment is lower than the compression rate of the encoding method corresponding to the second original data segment; encoding each original data segment in the original multimedia data according to the selected encoding method to generate encoded multimedia data corresponding to the original multimedia data.

[0038] In a possible design, it also includes: an image signal processor ISP, located between the sensor and the processor, and used to perform signal processing on the original multimedia data before sending the original multimedia data to the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a network architecture provided for an embodiment of the present application;

[0040] Figure 2 A flowchart of a multimedia data encoding method provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of identifying a valuable event in an image frame provided by an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0045] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0046] The embodiments of the present application can be applied to a video surveillance system. When a camera transmits captured video segments, image frames or other multimedia data such as image frames to a multimedia management device, the present application can identify multimedia data of different value levels to determine the time to send different multimedia data to the multimedia management device according to different value levels, so as to save bandwidth during multimedia data transmission.

[0047] refer to Figure 1 , Figure 1The network architecture provided for the present application may include a camera 10, a transmission system 11 and a storage system 12. The camera 10 may include a camera, which may be an analog camera, a network camera or a high-definition camera, etc. The camera may include a microphone, a sensor, an encoder and an acquisition card, etc., wherein the microphone may also be referred to as a combination of a monitoring head, a microphone and an amplifier, etc. The transmission system 11 may include transmission devices such as video cables, audio cables, wireless networks, network cables, optical fibers, optical transceivers, switches and routers. The storage system 12 includes a display device, a multimedia management device (storage device) and a control device. Among them, the display device may be a monitor or a large-screen TV wall, etc., the storage device may be a network video recorder (NVR), an Internet Protocol storage area network (IP SAN) or a cloud storage device, etc., and the control device may be a video distributor, a matrix, a control keyboard and an integrated management platform, etc.

[0048] The camera 10 is mainly used for collecting audio and video data. The transmission system 11 is used to transmit the audio and video signals collected by the camera and encoded and decoded to the storage system. The storage system 12 is used to display the audio and video signals received from the transmission device in real time on the display device or store them in the hard disk medium. It can also perform big data analysis or AI analysis on the audio and video signals to obtain valuable information, and can further optimize or classify the audio and video data for user viewing.

[0049] For example, in security scenarios, the camera can send the captured image frames or video segments to the storage system through the transmission system, and the image frames or video segments can be stored in the storage device of the storage system. The control device can read the data in the storage device and control the data to be distributed to multiple display devices, or select any display device to display the picture on the large-screen TV wall. In the process of data transmission, a large bandwidth is required, and when storing data, a large storage cost is also required.

[0050] The camera can be connected to the transmission system and the storage system by wire or by wireless connection.

[0051] It should be noted that in some scenarios, the original multimedia data acquisition end and the data receiving end, the multimedia management device (storage device) can be different devices. For example, in a security scenario, the acquisition end that captures the original multimedia data can include a camera, and the multimedia management device can include an NVR, which is the storage and forwarding part of the video surveillance system. The NVR works in conjunction with the network camera. In other scenarios, the original multimedia data acquisition end and the data receiving end can also be the same device. For example, smart phones, tablet computers, driving recorders, etc., have storage functions in addition to camera functions. In this scenario, the transmission system can be set to default.

[0052] In a video surveillance system, the camera captures video segments or image frames in real time, and generally stores the same amount of multimedia data content as it captures to a storage device. Among them, valuable data accounts for a small proportion, and most of the data of little value or worthlessness consumes a lot of transmission bandwidth and storage costs. Although currently, in order to save bandwidth, the camera can transmit the feature values ​​of video segments or image frames in real time, and cache the captured on-site photos locally in the camera, and asynchronously transmit them to the storage device when idle. For example, when there is an image area of ​​a person in the image frame captured by the camera, only the feature values ​​of the person's face, nose, eyes, mouth, ears and other senses can be sent to the storage device in real time, and the image frame can be temporarily stored locally in the camera, and the image frame can be transmitted to the storage device when the network is idle.

[0053] Although this solution relieves the pressure of transmitting video frames or image frames when the network is busy, it fails to reduce the amount of data transmitted, and therefore fails to reduce the storage space occupied by the storage system. In addition, the image frames are not transmitted to the storage system together with the feature values, which easily causes incomplete testimony data and affects the real-time decision-making of the back-end. For example, when the feature value matches the feature of a certain person pre-stored in the storage system, the person is a marked dangerous person, but the real-time image of the person is not transmitted to the storage system in time, so the environment where the person is at that time cannot be known in time, which brings inconvenience to the user. Moreover, when the picture frame is temporarily stored locally in the camera, the local storage space of the camera is limited in the security scene, so there is space recycling. For example, when the storage disk capacity reaches the recycling threshold, the camera can start deleting from the oldest file to ensure that the latest picture frame or video frame can be cached locally in the camera. However, there may be a risk that the picture frame or video frame at the scene is deleted locally in the camera before it is transmitted to the storage system, which is easy to cause a major accident of key data loss.

[0054] The present application aims at the problem that image frames or video frames cannot be transmitted to the storage system in time, resulting in the risk of incomplete or deleted supporting data, and proposes a multimedia data processing method. In this method, the camera still sends all multimedia data content to the storage device, but the camera can identify multimedia data of different value levels, so that multimedia data of different value levels can be encoded in different encoding methods, and different encoding methods correspond to different compression rates. For example, the compression rate of the encoding method corresponding to the multimedia data of high value level is low, and the compression rate of the encoding method corresponding to the multimedia data of low value level is high. In this way, for the multimedia data of high value level, it adopts a visual lossless encoding method with high fidelity. For the multimedia data of low value level, it adopts a visual lossy encoding method with low fidelity. In this way, when transmitting, the multimedia data of high value level occupies less bandwidth than the multimedia data of low value level due to its low compression rate, and when stored in the storage system, the multimedia data of high value level requires less storage space due to its low compression rate, and the multimedia data of low value level requires more storage space due to its high compression rate. In this way, on the one hand, a certain amount of transmission bandwidth can be saved, and a certain amount of storage space can also be saved.

[0055] The embodiments of the present application are further described below.

[0056] The present application embodiment provides a multimedia data encoding method, such as Figure 2 As shown, the method includes:

[0057] 201. The camera shoots and obtains original multimedia data.

[0058] The camera can shoot the environment within the viewing angle of the lens in real time to obtain original multimedia data, which can be a video file obtained when the camera shoots. The video file can be composed of multiple image frames. Image frames can also be understood as video frames.

[0059] The original multimedia data in this application can also be understood as video data captured by a camera before being encoded.

[0060] 202. The camera determines the value level of each original data segment in the original multimedia data according to the value level rule, the original multimedia data includes multiple original data segments, wherein the multiple original data segments include a first original data segment and a second original data segment, and the value level of the first original data segment is higher than the value level of the second original data segment.

[0061] Determining the value level of each original data segment in the original multimedia data according to the value level rule may include: identifying the value event in the original multimedia data through semantic analysis, and determining the value level of the original data segment where the value event is located according to the value event, wherein the value level rule is used to describe the corresponding relationship between the value event and the value level. It is also possible to correspond the region of interest (ROI) and non-ROI to different value levels, for example: the foreground part of the video / image is regarded as a high value level, and the background part is regarded as a low value level.

[0062] The semantic analysis can be AI algorithms such as computer vision, video semantic analysis, and machine learning. Among them, computer vision can be understood as: using a camera to replace the human eye to identify, track, and measure the target, and further perform graphic processing to make the camera process an image that is more suitable for human eye observation or transmission to instrument detection. Video semantic analysis can be understood as: establishing a mapping relationship between the low-level physical feature space and the high-level semantic space of the video through computer vision processing, extracting semantic information that can reflect human subjective concepts from the video content, and automatically annotating the semantic content of the video. Machine learning can be understood as a science of artificial intelligence. The main research object in this field is artificial intelligence. It uses data or past experience to automatically improve computer algorithms to optimize computer program performance.

[0063] That is to say, when the camera obtains the original multimedia data, it will perform value analysis on the original multimedia data to determine the value event in the original multimedia data, and then determine the value level corresponding to the value event according to the corresponding relationship between the value event and the value level in the value level rule. The original multimedia data includes multiple original data segments, and the value level corresponding to the value event is the value level corresponding to the original data segment where the value event is located.

[0064] The original data segment may include one or more of the following: a video segment, a complete image frame, a portion of an image frame (incomplete image frame). The original data segment may be a video segment, an image frame, or a portion of an image frame, which is determined by the value event of the original multimedia data.

[0065] In some embodiments, during the shooting process, the camera captures a video file composed of multiple image frames. The original multimedia data can be understood as a video file composed of multiple image frames. The camera identifies the value event of the original multimedia data, which can be understood as the camera identifying the value event of the video file. Since the video file is composed of multiple image frames, when the camera captures each image frame, the camera can analyze the image frame to determine the value event of the image frame. If the camera identifies the existence of a value event in the image frame through semantic analysis, the number of value events in the image frame can be determined. If there is at least one value event, the number of at least one value event in the image frame is less than or equal to the first threshold, then the camera can determine that a part of the image frame has a value event, so the camera can determine the value level corresponding to at least one value event in a part of the image frame according to the corresponding relationship. In this case, each value event in a part of the image frame can be a raw data segment. If the number of value events in the image frame is greater than the first threshold, the camera can use the entire area of ​​the image frame as a raw data segment. When the raw data segment is determined, the camera can determine the value level corresponding to the raw data segment where the value event is located according to the value rule level.

[0066] For example, when a camera captures a complete image frame, each object in the image frame can be identified by semantic analysis. Figure 3 , for example, the object includes one or more of the following: at least one person A, at least one animal B, at least one building C or at least one plant D, etc. When the camera recognizes these multiple objects, the multiple objects can be compared with multiple value events in the preset value event library to determine whether there are some objects among the multiple objects whose similarity with the preset multiple value events is higher than the second threshold. If there are objects with a similarity higher than the second threshold, for example, it is determined that the similarity between a certain person A and a preset value event is higher than the second threshold, the camera determines that there is a value event of person A in the image frame. The data of the portrait area of ​​person A in the image frame is an original data segment. If the first threshold is 3, then 3 or more objects whose similarity to the preset value event is higher than the second threshold are determined in the image frame, and the camera can use the image frame as a raw data segment, and when determining the value level, the value level of the image frame can be determined; if the number of objects whose similarity to the preset value event is higher than the second threshold is less than 3, and there are 2 objects in the image frame, the camera uses the data of the image area corresponding to each of the 2 objects as a raw data segment, for a total of 2 raw data segments, each of which records the value event. When determining the value level, the value level of the raw data segment of each object will be determined.

[0067] For example, in a security scene, the camera needs to determine in real time whether there is a specific tracking person among the objects within the captured viewing angle. Therefore, when the camera captures that the only person A among the objects in the currently captured image frame is a person, and the similarity between the portrait area of ​​person A and the value event E in the value event library is higher than the second threshold, the camera can use the image area other than person A in the image frame as the image area of ​​a non-value event. If the value event E describes the portrait of a criminal suspect, the camera can determine that person A is a criminal suspect. Since the value event in the image frame is only person A, and the objects of the value event are less than 3, the camera uses the portrait area of ​​person A in the image frame as a raw data segment, and then determines the value level of the raw data segment according to the corresponding relationship. At the same time, the camera determines the image area other than the portrait area of ​​person A in the image frame as a non-value event. Similarly, if there are multiple people in the image frame whose similarity with the specific tracking person is higher than the second threshold, and the number is greater than 3, the camera can determine the entire area of ​​the image frame as a raw data segment and determine the value level of the raw data segment.

[0068] When determining the similarity, the facial features of the person in the image frame can be semantically identified, and the facial features of the person can be compared with the facial features of the persons stored in the value event library, and the person whose similarity after comparison is higher than the second threshold can be determined as the tracked person.

[0069] In some embodiments, the original data segment can also be a video segment, that is, the camera can use a video segment composed of multiple image frames as an original data segment. In this case, if the camera determines that the similarity between a certain object in the first image frame and a preset value event is greater than or equal to the third threshold value and less than the second threshold value, the camera can determine that the object in the image frame is not very similar to the value event in the value event library, but there is a suspected object with a certain similarity, that is, the camera cannot accurately determine that the object is indeed a value event in the value event library. At this time, the camera can continue to track multiple image frames continuously captured after the first image frame, and use the video segment composed of the first image frame and multiple image frames as an original data segment to determine the value level of the video segment as an original data segment. In this case, as long as there is a suspected object in the first image frame, the video segment of the first image frame is used as an original data segment, and the suspected value event is recorded in the original data segment.

[0070] In some embodiments, if the threshold for determining the similarity with a valuable event includes a second threshold and a third threshold, and when the similarity between each object in the image frame and a valuable event in the valuable event library is less than the third threshold, the camera can determine the image frame as an original data segment, and the original data segment does not record a valuable event, and the image frame can be treated as a non-valuable event.

[0071] In summary, if the above-mentioned original multimedia data includes multiple original data segments, and the multiple original data segments include a first original data segment and a second original data segment, then when a valuable event is recorded in the first original data segment and no valuable event is recorded in the second original data segment, the value level of the first original data segment is higher than the value level of the second original data segment.

[0072] For example, taking the security scenario as an example, if the customer requires the video surveillance system to track multiple types of key personnel, such as criminal suspects, the camera side needs to monitor these key personnel in real time online to determine whether there are people who are highly similar to these key personnel within the camera's field of view, so as to promptly notify the user of the storage system. These key personnel can be pre-configured on the camera side, or the storage system or other system can configure the camera through the network, that is, by sending configuration information to the camera, so that the camera can determine the original data segment of the multimedia data, as well as the value event and value level corresponding to the original data segment according to the configuration information.

[0073] In some embodiments, referring to Table 1, the configuration information may include value events, event labels, value levels, and event coordinates. Value events may be images of objects that need to be tracked, such as portraits of multiple types of key personnel in security scenes. The camera may determine whether there are value events and suspected value events in the real-time captured image frames based on the configured value events, and events other than value events and suspected value events are determined as non-value events.

[0074] Event labels can be used to indicate the labels that the camera can use for the original data segment, for example, to indicate that the camera can use multiple specific types of key personnel to mark the objects that need to be tracked. For example, event labels can be used to indicate that when the camera determines that there is a valuable event, the event label that can be added to the original data segment where the valuable event is located can be the personnel type of the key personnel. For example, in a security scenario, if the camera determines that there is a valuable event with a high similarity to a criminal suspect in the image frame, the event label of the criminal suspect can be added to the original data segment of the valuable event; or, when the camera determines that there is a suspected valuable event suspected of a criminal suspect in the image frame, the event label of the suspected valuable event can be determined as suspected, and then the event labels of key personnel and events other than suspected can be determined as safe.

[0075] The value level may be the level of a value event, for example, the value level of multiple types of key personnel is high, the value level of suspected key personnel (suspected value events) is medium, and the value level of other personnel (non-value events) is low.

[0076] The event coordinates can indicate the target coordinates of the valuable events that the camera needs to mark in the image frame. The target coordinates can be understood as the image area of ​​the valuable event in the image frame. For example, the camera can use the event coordinates of key personnel to indicate the portrait area of ​​the key personnel in the image frame; use the event coordinates of suspected valuable events to indicate the portrait area of ​​suspected key personnel in the image frame, and mark the event coordinates of non-valuable events as default.

[0077] Table 1

[0078] event Value Level Event Tags Event Coordinates Value Events high Criminal suspect Target coordinates Suspected value event middle Suspected Target coordinates Non-value events Low Safety default

[0079] 203. The camera selects an encoding method corresponding to the value level of each of the multiple original data segments. Since encoding will cause a reduction in the amount of data, encoding can also be considered as compressing the data. Different encoding methods have different compression rates. The compression rate of the encoding method corresponding to the first original data segment is lower than the compression rate of the encoding method corresponding to the second original data segment.

[0080] It should be noted that different encoding algorithms often correspond to different encoding methods, but the encoding method is not the same as the encoding algorithm. For example, the same encoding algorithm may include multiple different levels, and different levels have different compression rates. Then each compression rate corresponds to an encoding method.

[0081] When the value level of the original data segment is determined, the camera can determine the corresponding encoding method according to the value level. When the value level of the first original data segment is higher than the value level of the second original data segment, the camera can determine that the compression rate of the encoding method corresponding to the first original data segment is lower than the compression rate of the encoding method corresponding to the second original data segment.

[0082] In some embodiments, when a valuable event is recorded in the first raw data segment, or when a suspected valuable event is recorded, the camera may use a low compression rate, or a compression rate that can ensure visual losslessness, or a high fidelity compression rate to compress the data. When no valuable event is recorded in the second raw data segment, or when the events in the second raw data segment are all non-valuable events, the camera may use a high compression rate, or a visual lossy, or a low fidelity compression rate to compress the data.

[0083] Exemplarily, corresponding to the implementation of Table 1, referring to Table 2, different compression rates can be used according to different value levels. For example, for a value event with a high value level, the original data segment corresponding to the value event can be encoded in a low compression rate, and the original data segment can correspond to the value event area of ​​the image frame, and the value event area is a partial area or the entire area of ​​the image frame, or the original data segment can correspond to a video segment. The corresponding encoding method, for example, uses high-definition compression or lossless compression for the value event area of ​​the image frame, and standard compression for the area outside the value event area of ​​the image frame; or when the value event is a video segment, high-definition compression or lossless compression is used for the video segment.

[0084] For suspected value events with a medium value level, the encoding method of the original data segment of the suspected value event can also be similar to the encoding method of the value event with a high value level, that is, high-definition compression or lossless compression is used for the suspected value event area in the image frame, and standard compression is used for the area outside the suspected value event area in the image frame. When the suspected value event is a video segment, high-definition compression or lossless compression can be used for the video segment.

[0085] For events with low value levels, or for non-value events, a high-compression-rate low-definition compression (lossy compression) can be used for the original data segment of the non-value event. In this case, no definition requirement is made for the image frame or video segment.

[0086] Among them, for example, for the encoding method, high-definition compression can be high-efficiency video coding (HEVC), and lossless compression can be the moving picture experts group (MPEG) standard. Standard compression can be the encoding methods such as H.261, H.263 or H.264 proposed by the International Telecommunication Union. Low-definition compression can be, for example, the joint photographic experts group (JPEG).

[0087] Table 2

[0088] Value Level Data content Encoding high Image frame value event area High-definition compression or lossless compression middle Suspected value event area of ​​the image frame Standard compression Low Image frame worthless area Low-resolution compression.

[0089] In the above embodiment, when the valuable event is in the image frame, the valuable event area can be encoded in a low compression rate, and the non-value event area can be encoded in a high compression rate; or the video segment where the valuable event is located can be encoded in a low compression rate. In some other embodiments, the encoding method of the same image frame can also be distinguished and encoded according to other distinguishing methods. For example, for the same image frame, there may be multiple pedestrians. In the embodiment of the present application, when a pedestrian who needs to be paid attention to is detected from multiple pedestrians through semantic recognition (for example, it can be at least one of the above-mentioned key personnel), when determining the value level, the pedestrian who needs to be paid attention to in the image frame can be determined as a valuable event, and the image area of ​​the pedestrian who needs to be paid attention to in the image frame is a raw data segment, which can be encoded in a low compression rate, such as HEVC; the pedestrian who does not need to be paid attention to in the image frame can be determined as a non-value event, but because the non-value event is a pedestrian, the raw data segment of the image area in the image frame can be encoded in a compression rate of medium, such as the MPEG standard; the background area other than the pedestrian in the image frame can be encoded in a high compression rate, such as JPEG.

[0090] Among them, when the camera captures multiple pedestrians, the facial features of each pedestrian can be extracted. After comparing the extracted facial features with the facial features of the people stored in the value event library, pedestrians with a similarity higher than a certain threshold can be determined as pedestrians that need attention, pedestrians with a similarity lower than the threshold can be determined as pedestrians that do not need attention, and the image area outside the multiple pedestrians can be determined as the background area.

[0091] It should be noted that the encoding method corresponding to the value level may be pre-configured in the camera, or may be configured to the camera by the storage system through the network. For example, the above configuration information may also include the encoding method corresponding to the value level.

[0092] 204. The camera encodes each original data segment in the original multimedia data according to the selected encoding method to generate encoded multimedia data corresponding to the original multimedia data.

[0093] The camera may obtain raw multimedia data within a viewing angle through a sensor therein. The format of the raw multimedia data generated by the sensor, or the format of each generated image frame, may vary depending on the design of the sensor.

[0094] In some embodiments, the format of the raw multimedia data generated by the sensor may be in RAW format, and the RAW format may be different depending on the design of the sensor. For example, the RAW format may be one of a variety of formats such as bayer RGGB, RYYB, RCCC, RCCB, RGBW, CMYW, etc. The camera may also use image signal processing (ISP) to convert the RAW format acquired by the sensor into one of the following formats: YUV format, HSV format, Lab format, CMY format, or YCbCr format. In each embodiment of the present application, the original multimedia data is still referred to as raw multimedia data after being processed by ISP, that is, the format of the raw multimedia data may also be one of YUV format, HSV format, Lab format, CMY format, or YCbCr format.

[0095] When each raw data segment in the raw multimedia data is encoded by the encoder in the camera, the image frame can be converted into encoded multimedia data that is easy for the human eye to recognize and occupies less storage space. The encoded multimedia data can be an image media format or a video media format. Among them, the image media format can be, for example, one of the following formats: jpeg format, bmp format, tga format, png format, and gif format; the video media format can be one of the following formats: MPEG format, AVI format, nAVI format, ASF format, MOV format, WMV format, 3GP format, RM format, RMVB format, FLV / F4V format, H.264 format, or H.265 format.

[0096] Therefore, when the camera obtains the encoded multimedia data, the encoded multimedia data can be sent to the storage device through the transmission system for storage.

[0097] In summary, the present application can divide the original multimedia data into original data segments of different value levels, and can use the encoding methods corresponding to the different value levels to encode the original data segments, using the encoding method with a low compression rate for the original data segments with high value levels, and using the encoding method with a high compression rate for the original data with low value levels. This can not only ensure that the encoded multimedia data after encoding the original data with high value levels can be transmitted back to the back-end device in time, so that the user can know the movements of key personnel or concerned personnel in time. In addition, since the data sizes corresponding to different compression rates are different and the storage spaces occupied are different, the storage system can achieve the effect of saving the storage space of the storage device when storing the encoded multimedia data.

[0098] In some embodiments, when the encoded multimedia data is sent to the storage device, it can be sent according to different situations. In the embodiments of the present application, the original multimedia data can include a first original data segment with a high value level and a second original data segment with a lower value level, and the first original data segment includes a value event, that is, a person who needs to be paid attention to or a key person, therefore, the first original data segment can be immediately sent to the multimedia management device, or in other words, when the first original data segment is encoded to obtain the encoded multimedia data, the encoded multimedia data obtained after encoding can be immediately sent to the multimedia management device, and the multimedia management device can be a storage device of the storage system, so that the manager of the storage system can obtain the movement of the person who needs to be paid attention to or the key person in a timely manner. The value level of the second original data segment is relatively low, and the second original data segment does not include any value events. Therefore, the second original data segment can be temporarily stored in the local non-volatile storage medium of the camera (such as a disk, a solid-state hard disk, or an erasable optical disk) so that it can be read from the non-volatile storage medium and sent to the back-end multimedia management device when the camera is idle; or the encoded multimedia data obtained after encoding the second original data segment can be temporarily stored in the local non-volatile storage medium of the camera so that it can be read from the non-volatile storage medium and sent to the back-end multimedia management device when the camera is idle.

[0099] In some embodiments, for the encoded multimedia data corresponding to the first original data segment with a high value level, when it is immediately transmitted to the multimedia management device of the storage system, the encoded multimedia data corresponding to the first original data segment can be sent to the multimedia management device using a high-speed network, so that the manager of the storage system can timely obtain important information recorded in the first original data segment, such as criminal suspect information. For the encoded multimedia data corresponding to the second original data segment with a low value level, when it is read locally from the camera when the camera is idle and sent to the multimedia management device of the storage system, the encoded multimedia data corresponding to the second original data segment can be sent to the multimedia management device using a low-speed network. In this way, the timely transmission of the data segment with a high value level can be guaranteed, and the bandwidth pressure during network transmission can also be reduced.

[0100] Exemplarily, referring to Table 3, assuming that the value level includes three levels: high, medium and low, when the original data segment with a high value level is determined, the encoded multimedia data after encoding the original data segment can be immediately sent to the multimedia management device and sent using a high-speed network. The high-speed network can be, for example, a highly reliable transmission protocol, such as 5G wireless transmission, or ultra-reliable communication (enhanced machine type communication, eMTC) slice transmission. When the original data segment with a medium value level is determined, a low-speed network transmission can be used, for example, the encoded multimedia data after encoding the original data segment is transmitted using an enhanced mobile broadband (Enhanced Mobile Broadband, eMBB) large-capacity demand network slice. When the original data segment with a low value level is determined, the original data segment can be temporarily stored in the camera locally, and when the camera's network is idle, the encoded multimedia data after encoding the original data segment can be transmitted using a low-speed network, for example, using an eMBB large-capacity demand network slice to transmit to the multimedia management device.

[0101] Table 3

[0102]

[0103]

[0104] When the multimedia management device stores the encoded multimedia data, the encoded multimedia data corresponding to different value levels may use different storage media and storage periods.

[0105] In some embodiments, for the first original data segment with a high value level, a storage medium with a long-term storage can be used; for the second original data segment with a low value level, a storage medium suitable for caching can be used. In this way, for important data, a storage medium with a longer storage period can be used so that the administrator can conveniently view the data when needed. For non-important data, caching can be performed so that the newly acquired data can be stored in a cacheable storage medium in a timely manner.

[0106] Exemplarily, referring to Table 4, assuming that the value level includes three levels: high, medium and low, when the multimedia management device determines that it has received the encoded multimedia data corresponding to the original data segment with a high value level, the encoded multimedia data can be stored in a storage medium that can be preserved for a long time. For example, the storage medium can be a storage class memory (SCM) or a Blu-ray disc storage medium, or a solid state disk (SSD), or a disk array that constitutes a redundant array of independent disks (RAID) 1. Among them, SCM not only enjoys the performance of dynamic random access memory (DRAM), but also has the capacity advantage of NAND Flash, and has the commonality of storage-level persistence and memory fast byte-level access. The storage period can be 90 days or permanent.

[0107] When the multimedia management device determines that the encoded multimedia data corresponding to the original data segment with a value level of medium is received, the encoded multimedia data can be stored in a general video or image storage medium. For example, the general video or image storage medium can be a common serial ATA (serial advanced technology attachment, SATA) hard disk, also known as a serial port hard disk, or a disk array constituting RAID5. The storage period can be 30 days, or other time periods.

[0108] For the encoded multimedia data corresponding to the original data segments of low value level, a storage medium suitable for temporary storage and frequently erasable can be used to cache the data locally in the camera. The storage period can be 14 days, or other time periods.

[0109] Table 4

[0110]

[0111] Therefore, in the processing method for multimedia data in the embodiment of the present application, original data segments of different value levels can be determined for the multimedia data, and the encoding methods corresponding to the original data segments of different value levels can be different. The compression rate of the original data segment with a high value level can be lower than the compression rate of the original data segment with a low value level. In this way, the original data segment with a high value level can be encoded using a high-fidelity encoding method, so that the multimedia data with a high value level can be played visually lossless in the multimedia management device at the back end. At the same time, for the original data segment with a high value level, a high-speed network can be used to immediately transmit it to the multimedia management device at the back end, and for the original data segment with a low value level, it can be temporarily stored in the local camera, and when the camera network is idle, it can be transmitted to the multimedia management device at the back end using a low-speed network. On the one hand, it can ensure that the data with a high value level is transmitted to the multimedia management device in time, and on the other hand, it can also alleviate the bandwidth pressure during the transmission of multimedia data. In addition, caching the data with a low value level to the local camera can also save the storage cost on the multimedia management device side.

[0112] Figure 4 The schematic block diagram of the electronic device 40 provided in the embodiment of the present application can be used to perform the above-mentioned method of multimedia data encoding. The electronic device 40 can be the camera in the above-mentioned embodiment. The electronic device 40 can include:

[0113] The shooting module 401 is used to shoot and obtain original multimedia data;

[0114] The processing module 402 is further configured to determine the value level of each original data segment in the original multimedia data according to the value level rule, wherein the original multimedia data includes a plurality of original data segments, wherein the plurality of original data segments include a first original data segment and a second original data segment, and the value level of the first original data segment is higher than the value level of the second original data segment;

[0115] The processing module 402 is further configured to select an encoding method corresponding to a value level of each of the plurality of original data segments, wherein different encoding methods correspond to different compression rates, and the compression rate of the encoding method corresponding to the first original data segment is lower than the compression rate of the encoding method corresponding to the second original data segment;

[0116] The encoding module 403 is further configured to encode each original data segment in the original multimedia data according to the selected encoding method to generate encoded multimedia data corresponding to the original multimedia data.

[0117] In the embodiment of the present application, the original data segment includes one or more of the following: a video segment, a complete image frame, and a portion of an image frame.

[0118] In an embodiment of the present application, the processing module 402 can be used to: identify value events in the original multimedia data through semantic analysis, and determine the value level of the original data segment where the value event is located according to the value event, wherein the value level rule is used to describe the correspondence between the value event and the value level.

[0119] In the embodiment of the present application, valuable events are recorded in the first original data segment; and valuable events are not recorded in the second original data segment.

[0120] In an embodiment of the present application, the electronic device 40 also includes a transceiver module 404, which is used to: immediately send the first original data segment to the multimedia management device; the electronic device 40 also includes a storage module 405, which is used to store the second original data segment in the local non-volatile storage medium of the camera, and then send the second original data segment in the non-volatile storage medium to the multimedia management device.

[0121] In the embodiment of the present application, it also includes a transceiver module 404, which is used to: send the first original data segment to the multimedia management device using a high-speed network; and send the second original data segment to the multimedia management device using a low-speed network.

[0122] In an embodiment of the present application, the format of the original multimedia data includes: one of the RAW format, RGB format or YUV format, HSV format, Lab format, CMY format or YCbCr format; the format of the encoded multimedia data includes: one of the image media format or video media format.

[0123] It should be understood that the electronic device 40 according to the embodiment of the present application may correspond to the camera in the method embodiment of the present application, and the operations and / or functions of the various modules in the electronic device 40 are respectively to achieve Figure 2 For the sake of brevity, the corresponding process of the method will not be repeated here.

[0124] like Figure 5 FIG. 5 is a schematic diagram of a hardware architecture of an exemplary image processing device provided in an embodiment of the present application. The hardware architecture of the image processing device 50 is applicable to Figure 2 Camera in.

[0125] Exemplarily, the image processing device 50 may include a lens assembly connected via a memory bus, at least one central processing unit (CPU), at least one memory, a pickup, a sensor, a GPU, an encoder, a receiving interface, and a sending interface. Optionally, the image processing device 50 may also include a decoder (not shown in the figure), a dedicated video or graphics processor, a microprocessor and a microcontroller unit (MCU) (not shown in the figure), etc. In an optional case, the above-mentioned various parts of the image processing device 50 are coupled through a connector. It should be understood that in various embodiments of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines or buses, etc. These interfaces are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces, and this embodiment does not limit this. In an optional case, the above-mentioned various parts are integrated on the same chip; in another optional case, the CPU, GPU, encoder, receiving interface and sending interface are integrated on one chip, and the various parts inside the chip access the external memory through the bus. The dedicated video / graphics processor can be integrated with the CPU on the same chip, or it can exist as a separate processor chip. For example, the dedicated video / graphics processor can be a dedicated ISP. The ISP can be located between the sensor and the processor, and is used to perform signal processing on the original multimedia data before sending the original multimedia data to the CPU. The chip involved in the embodiment of the present application is a system manufactured on the same semiconductor substrate using an integrated circuit process, also called a semiconductor chip, which can be a collection of integrated circuits formed on the substrate (usually a semiconductor material such as silicon) using an integrated circuit process, and its outer layer is usually encapsulated by a semiconductor packaging material. The integrated circuit may include various functional devices, each of which includes transistors such as logic gates, metal oxide semiconductor (MOS) transistors, bipolar transistors or diodes, and may also include other components such as capacitors, resistors or inductors. Each functional device can work independently or under the action of necessary driver software, and can realize various functions such as communication, calculation, or storage.

[0126] In some embodiments, the CPU may be a single-CPU processor or a multi-CPU processor; alternatively, the CPU may be a processor group consisting of multiple processors, and the multiple processors are coupled to each other through one or more buses. In an optional case, the processing of the image signal or video signal is partially completed by the GPU, partially completed by a dedicated video / graphics processor, and possibly completed by software code running on a general-purpose CPU or GPU.

[0127] and Figure 4 Correspondingly, the CPU can be Figure 4 The processing module 402 in the electronic device has the same function.

[0128] The memory can be used to store computer program instructions, including an operating system (OS), various user applications, and various types of computer program codes including program codes for executing the scheme of the present application; the memory can also be used to store video data, image signal data, etc.; the CPU can be used to execute the computer program code stored in the memory to implement the method in the embodiment of the present application. Optionally, the memory may be a non-power-off volatile memory, such as an embedded multi media card (EMMC), universal flash storage (UFS) or read-only memory (ROM), or other types of static storage devices that can store static information and instructions. It may also be a power-off volatile memory (volatile memory), such as a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions. It may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other computer-readable storage medium that can be used to carry or store program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0129] and Figure 4 Correspondingly, the memory can be Figure 4 The storage module 405 in has the same function.

[0130] The receiving interface may be an interface for data input of a processor chip, and in an optional case, the receiving interface may be a high definition multimedia interface (HDMI). The sending interface may be an interface for data reception of a processor chip.

[0131] The receiving interface and the sending interface can be Figure 4 The transceiver module 404 in has the same function.

[0132] The encoder can be with Figure 4 The encoding module 403 in has the same function.

[0133] The sensor can be Figure 4 The shooting module 401 in the embodiment has the same function and can convert the light received by the lens assembly into original multimedia data.

[0134] It should be understood that in various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. Through the description of the above implementation mode, technicians in the relevant field can understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0139] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multimedia data encoding method, characterized in that: include: The camera shoots and obtains original multimedia data; Determine the value level of each original data segment in the original multimedia data according to the value level rule, the original multimedia data includes multiple original data segments, wherein the multiple original data segments include a first original data segment and a second original data segment, and the value level of the first original data segment is higher than the value level of the second original data segment; wherein the original multimedia data is a video file composed of multiple image frames, if the number of value events in the image frame is less than or equal to a first threshold, then the image area corresponding to each value event in the image frame is one of the first original data segments, and the image area other than the image area corresponding to each value event in the image frame is the second original data segment; if the number of value events in the image frame is greater than the first threshold, then the entire area of ​​the image frame is one of the first original data segments; the value event is an image of an object to be tracked; selecting an encoding method corresponding to the value level of each of the plurality of original data segments, wherein different encoding methods correspond to different compression rates, and the compression rate of the encoding method corresponding to the first original data segment is lower than the compression rate of the encoding method corresponding to the second original data segment; Each original data segment in the original multimedia data is encoded according to the selected encoding method to generate encoded multimedia data corresponding to the original multimedia data.

2. The method according to claim 1, characterized in that Determining the value level of each original data segment in the original multimedia data according to the value level rule specifically includes: The value events in the original multimedia data are identified through semantic analysis, and the value level of the original data segment where the value event is located is determined according to the value event, wherein the value level rule is used to describe the corresponding relationship between the value event and the value level.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Immediately sending the first original data segment to the multimedia management device; The second original data segment is stored in the local non-volatile storage medium of the camera, and then the second original data segment in the non-volatile storage medium is sent to the multimedia management device.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: Sending the first original data segment to the multimedia management device via a high-speed network; The second original data segment is sent to the multimedia management device using a low-speed network.

5. The method according to claim 1 or 2, characterized in that: The format of the original multimedia data includes: one of RAW format, RGB format, YUV format, HSV format, Lab format, CMY format or YCbCr format; The format of the encoded multimedia data includes: an image media format or a video media format.

6. An electronic device, characterized in that: include: A shooting module, used for shooting and obtaining original multimedia data; The processing module is further used to determine the value level of each original data segment in the original multimedia data according to the value level rule, wherein the original multimedia data includes multiple original data segments, wherein the multiple original data segments include a first original data segment and a second original data segment, and the value level of the first original data segment is higher than the value level of the second original data segment; wherein the original multimedia data is a video file composed of multiple image frames, if the number of value events in the image frame is less than or equal to a first threshold, then the image area corresponding to each value event in the image frame is one of the first original data segments, and the image area other than the image area corresponding to each value event in the image frame is the second original data segment; if the number of value events in the image frame is greater than the first threshold, then the entire area of ​​the image frame is one of the first original data segments; the value event is an image of an object to be tracked; The processing module is further configured to select an encoding method corresponding to a value level of each of the plurality of original data segments, wherein different encoding methods correspond to different compression rates, and the compression rate of the encoding method corresponding to the first original data segment is lower than the compression rate of the encoding method corresponding to the second original data segment; The encoding module is further used to encode each original data segment in the original multimedia data according to the selected encoding method to generate encoded multimedia data corresponding to the original multimedia data.

7. The electronic device according to claim 6, characterized in that: The processing module is used to: The value events in the original multimedia data are identified through semantic analysis, and the value level of the original data segment where the value event is located is determined according to the value event, wherein the value level rule is used to describe the corresponding relationship between the value event and the value level.

8. The electronic device according to claim 6 or 7, characterized in that: The electronic device further comprises a transceiver module, which is used to: immediately send the first original data segment to the multimedia management device; The electronic device further comprises a storage module for storing the second original data segment in a local non-volatile storage medium of the electronic device, and then sending the second original data segment in the non-volatile storage medium to the multimedia management device.

9. The electronic device according to claim 6 or 7, characterized in that: Also includes a transceiver module for: Sending the first original data segment to the multimedia management device via a high-speed network; The second original data segment is sent to the multimedia management device using a low-speed network.

10. The electronic device according to claim 6 or 7, characterized in that: The format of the original multimedia data includes: one of RAW format, RGB format, YUV format, HSV format, Lab format, CMY format or YCbCr format; The format of the encoded multimedia data includes: an image media format or a video media format.

11. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 5.

12. A computer program product, characterized in that When the computer program product is executed on a computer, the electronic device is enabled to execute the method according to any one of claims 1 to 5.

13. A camera, characterized in that: include: A lens assembly, for receiving light; A sensor, configured to convert the light received by the lens assembly into raw multimedia data; Processor for: Determine the value level of each original data segment in the original multimedia data according to the value level rule, the original multimedia data includes multiple original data segments, wherein the multiple original data segments include a first original data segment and a second original data segment, and the value level of the first original data segment is higher than the value level of the second original data segment; wherein the original multimedia data is a video file composed of multiple image frames, if the number of value events in the image frame is less than or equal to a first threshold, then the image area corresponding to each value event in the image frame is one of the first original data segments, and the image area other than the image area corresponding to each value event in the image frame is the second original data segment; if the number of value events in the image frame is greater than the first threshold, then the entire area of ​​the image frame is one of the first original data segments; the value event is an image of an object to be tracked; selecting an encoding method corresponding to the value level of each of the plurality of original data segments, wherein different encoding methods correspond to different compression rates, and the compression rate of the encoding method corresponding to the first original data segment is lower than the compression rate of the encoding method corresponding to the second original data segment; Each original data segment in the original multimedia data is encoded according to the selected encoding method to generate encoded multimedia data corresponding to the original multimedia data.

14. The camera according to claim 13, characterized in that Also includes: An image signal processor ISP is located between the sensor and the processor, and is used to perform signal processing on the original multimedia data before sending the original multimedia data to the processor.

Images