A multimedia data processing method, device, electronic device and storage medium
Through image frame recognition and data processing priority allocation, desensitization of multimedia data in intelligent transportation systems is realized, the problem of data sensitive information protection is solved, and processing efficiency and response capabilities are improved.
Patent Information
- Application Number
- CN202411179188.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-27
AI Technical Summary
In intelligent transportation systems and autonomous driving technology, the multimedia data collected by vehicles contains a large amount of sensitive information. How to effectively desensitize to protect this information has become an urgent problem.
By acquiring image frames in the multimedia data, image recognition is performed to determine the target image frame, data processing priority is determined based on the target image frame, and multimedia data is desensitized by techniques such as mosaic processing and sampling frequency to form desensitized multimedia data.
Effectively protect sensitive objects in multimedia data, prevent information leakage, optimize data desensitization algorithms and processing processes, improve vehicle-side processing capabilities, reduce cloud pressure, and can quickly respond to environmental changes to ensure real-time processing of sensitive information.
Smart Images

Figure CN118709207B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of multimedia data processing, and in particular to a multimedia data processing method, device, electronic device and storage medium. Background Art
[0002] With the development of intelligent transportation systems and autonomous driving technology, more and more sensors and cameras are integrated into vehicles to collect multimedia data of the vehicle's surrounding environment in real time. These multimedia data often contain a large amount of sensitive information. Therefore, how to desensitize the collected multimedia data to protect sensitive information has become a technical problem that needs to be solved urgently. Summary of the invention
[0003] Embodiments of the present application provide a multimedia data processing method, device, electronic device, and storage medium.
[0004] According to a first aspect of the present application, a multimedia data processing method is provided, the method comprising: acquiring multiple image frames in multimedia data, the multimedia data being obtained by a data acquisition device on a target object performing data acquisition on the outside of the target object; performing image recognition on the multiple image frames to obtain a target image frame; the target image frame characterizes that an abnormality exists outside the target object; based on the target image frame, determining a data processing priority for each image frame; based on the data processing priority for each image frame, performing data desensitization on the multimedia data to obtain desensitized multimedia data.
[0005] According to one embodiment of the present application, performing image recognition on the multiple image frames to obtain a target image frame includes: obtaining a first image frame and a second image frame from the multiple image frames; the display time of the second image frame is after the first image frame; based on the first image frame and the second image frame, determining a change region in the second image frame; the change region characterizes a change between the first image frame and the second image frame; in response to the change region satisfying a set environmental mutation condition, determining the second image frame as the target image frame.
[0006] According to one embodiment of the present application, performing image recognition on the multiple image frames to obtain a target image frame includes: performing feature extraction on the image frames to obtain image features in the image frames; obtaining set reference features; the reference features characterize image features corresponding to special entities in the image frames; determining the similarity between the image features in the image frames and the reference features; and in response to the similarity being greater than a set similarity threshold, determining the corresponding image frame as the target image frame.
[0007] According to one embodiment of the present application, the data processing priority of each image frame is determined based on the target image frame, including: determining the time range corresponding to the target image frame based on the display time of the target image frame and the attribute characteristics of the target image frame; the time range at least includes the display time of the target image frame; determining the data processing priority of the image frames within the time range as a first priority; determining the data processing priority of the image frames outside the time range as a second priority; the first priority is higher than the second priority.
[0008] According to an embodiment of the present application, the multimedia data is desensitized based on the data processing priority of each image frame to obtain the desensitized multimedia data, including: performing target detection on the image frames of the first priority to obtain sensitive objects included in the image frames of the first priority; determining the pixel coordinates of the sensitive objects included in the image frames of the first priority; and performing mosaic processing on the image frames of the first priority based on the pixel coordinates to obtain the desensitized multimedia data.
[0009] According to an embodiment of the present application, the multimedia data is desensitized based on the data processing priority of each image frame to obtain desensitized multimedia data, including: obtaining the frame rate corresponding to the image frame of the second priority in the multimedia data; determining the sampling frequency based on the frame rate corresponding to the image frame of the second priority; extracting at least one third image frame from the image frame of the second priority based on the sampling frequency; in response to the third image frame satisfying the set data desensitization condition, performing data desensitization on the image frame of the second priority based on the third image frame to obtain the desensitized multimedia data.
[0010] According to one embodiment of the present application, the method also includes: obtaining a data transmission mode of the multimedia data; based on the data transmission mode, determining a corresponding multimedia encoding rule; based on the determined multimedia encoding rule, encoding the image frames in the desensitized multimedia data to obtain streaming multimedia data; based on the data transmission mode, transmitting the streaming multimedia data to a receiving end.
[0011] According to a second aspect of the present application, a multimedia data processing device is provided, the device comprising: an acquisition module, used to acquire multiple image frames in multimedia data, the multimedia data being obtained by a data acquisition device on a target object performing data acquisition on the outside of the target object; an identification module, used to perform image recognition on the multiple image frames to obtain a target image frame; the target image frame represents that an abnormality exists outside the target object; a determination module, used to determine a data processing priority of each image frame based on the target image frame; and a processing module, used to perform data desensitization on the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data.
[0012] According to a third aspect of the present application, an electronic device is provided, capable of any of the methods described above, including: a data acquisition unit, used to acquire multiple image frames in multimedia data, wherein the multimedia data is obtained by a data acquisition device on a target object performing data acquisition on the outside of the target object; an image recognition unit, used to perform image recognition on the multiple image frames to obtain a target image frame; the target image frame represents that there is an abnormality outside the target object; a priority determination unit, used to determine the data processing priority of each image frame based on the target image frame; and a data processing unit, used to perform data desensitization on the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data.
[0013] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the present application.
[0014] The method of the embodiment of the present application obtains multiple image frames in multimedia data, wherein the multimedia data is obtained by a data acquisition device on a target object performing data acquisition on the outside of the target object; image recognition is performed on the multiple image frames to obtain a target image frame; the target image frame indicates that there is an abnormality outside the target object; based on the target image frame, the data processing priority of each image frame is determined; based on the data processing priority of each image frame, the multimedia data is desensitized to obtain desensitized multimedia data. Through the present application, the collected multimedia data can be desensitized to protect sensitive information.
[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:
[0017] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.
[0018] Figure 1 The processing flow diagram of the multimedia data processing method provided in the embodiment of the present application is shown as follows Figure 1 ;
[0019] Figure 2 The processing flow diagram of the multimedia data processing method provided in the embodiment of the present application is shown as follows Figure 2 ;
[0020] Figure 3 The processing flow diagram of the multimedia data processing method provided in the embodiment of the present application is shown as follows Figure 3 ;
[0021] Figure 4 The processing flow diagram of the multimedia data processing method provided in the embodiment of the present application is shown as follows Figure 4 ;
[0022] Figure 5 The processing flow diagram of the multimedia data processing method provided in the embodiment of the present application is shown as follows Figure 5 ;
[0023] Figure 6 The processing flow diagram of the multimedia data processing method provided in the embodiment of the present application is shown as follows Figure 6 ;
[0024] Figure 7 The processing flow diagram of the multimedia data processing method provided in the embodiment of the present application is shown as follows Figure 7 ;
[0025] Figure 8 An optional schematic diagram of a multimedia data processing device provided in an embodiment of the present application is shown;
[0026] Fig. 9 An optional schematic diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0027] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0028] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0029] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0031] The processing flow of the multimedia data processing method provided in the embodiment of the present application is described. Figure 1 , Figure 1 The processing flow diagram of the multimedia data processing method provided in the embodiment of the present application is as follows Figure 1 , will combine Figure 1 Steps S101-S104 are shown for explanation.
[0032] Step S101 , obtaining a plurality of image frames in multimedia data, where the multimedia data is obtained by a data acquisition device on a target object collecting data outside the target object.
[0033] In some embodiments, the target object may be a vehicle, or other means of transportation, which is not limited in the embodiments of the present application. The data acquisition device may be a camera, and the data acquisition device may be arranged on the side of the target object. For example, one or more cameras are arranged on the side of the vehicle for collecting multiple image frames in multimedia data. An image frame may include a single image frame collected at a certain time node. Multimedia data may include a comprehensive data form that combines multiple types of data such as text, images, audio, and video, and the multimedia data may be an analog signal or a digital signal. The vehicle-mounted equipment on the vehicle may obtain multiple image frames in the multimedia data collected by the data acquisition device.
[0034] Step S102, performing image recognition on multiple image frames to obtain a target image frame; the target image frame indicates that an abnormality exists outside the target object.
[0035] In some embodiments, the target image frame may include an image frame that is identified as containing a sudden change in the environment or a special entity during the image recognition process. Image recognition may include: identifying the content in the image frame based on computer vision technology and classifying or marking the content in the image frame, wherein the content in the image frame may include objects, scenes, and activities, etc., and the embodiments of the present application do not limit the content in the specific image frame. The sudden change in the environment may include: scenes such as vehicles entering roundabouts, ramps, and intersections. The sudden change in the environment may also include: scenes such as construction on the road or accidents on the road. The sudden change in the environment may also include: scenes where the weather of the environment outside the target object changes. The sudden change in the environment may also include other scenes, which are not limited in the embodiments of the present application. Special entities may include: entities such as special buildings or special vehicles included in the image frame. Among them, special buildings may include management areas; special vehicles may include fire trucks. Special entities may also include other entities, which are not limited in the embodiments of the present application.
[0036] Step S103: determining the data processing priority of each image frame based on the target image frame.
[0037] Step S104: desensitizing the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data.
[0038] In some embodiments, data processing priority can be used in data desensitization tasks for multiple image frames to characterize the resources or attention corresponding to each data desensitization task. The higher the data processing priority of an image frame, the more resources or attention are allocated. The data processing priorities between each image frame can be the same or different. Data desensitization can characterize the process of removing or modifying sensitive information from multimedia data to protect privacy or ensure security. Desensitized multimedia data may include: multimedia data that has been processed with data desensitization, in which the sensitive information of each image frame in the multimedia data has been processed.
[0039] As an example, in a scenario where multimedia data is subjected to data desensitization processing, the target image frame and the frames adjacent to the target image frame are determined to have a higher data processing priority because they contain sudden environmental changes or special entities. Other image frames in the multimedia data are determined to have a lower data processing priority. For higher data processing priorities, the target image frame and the frames adjacent to the target image frame are subjected to data desensitization processing frame by frame. For lower data processing priorities, image frames are randomly selected from other image frames for data desensitization processing.
[0040] The method of the embodiment of the present application effectively protects sensitive objects in multimedia data and prevents information leakage of sensitive objects. The present application optimizes the data desensitization algorithm and processing flow, improves the vehicle-side processing capability, reduces the pressure on the cloud, can quickly respond to environmental changes, and ensures real-time processing of sensitive information. Through image frame recognition and random sampling, computing resources are reasonably allocated, and the data desensitization processing efficiency of multimedia data is improved.
[0041] In some embodiments, the processing flow of the multimedia data processing method is shown as follows: Figure 2 ,like Figure 2 As shown, performing image recognition on multiple image frames to obtain a target image frame in step S102 may include:
[0042] Step S201: obtaining a first image frame and a second image frame from a plurality of image frames.
[0043] Step S202: determining a changed area in the second image frame based on the first image frame and the second image frame.
[0044] In this embodiment, the display time of the second image frame is after the first image frame. The multimedia data may be composed of continuous image frames. The second image frame and the first image frame may include: two continuous or spaced image frames that appear sequentially in the video sequence. The change region characterizes the change between the first image frame and the second image frame. The change region can be obtained by identifying the first image frame and the second image frame through an image processing algorithm (such as a difference method). The embodiment of the present application does not limit the specific algorithm for determining the change region in the image frame.
[0045] Step S203 , in response to the change region satisfying the set environmental mutation condition, determining the second image frame as the target image frame.
[0046] In this embodiment, the environmental mutation condition may include a pre-set mutation condition or a threshold value. The environmental mutation condition may be used to determine whether the changes occurring in the change area are significant enough to be considered as the presence of an environmental mutation in the image frame. The environmental mutation condition may include: the size of the change area, the location of the change area, the speed of change, and the change in color space. The environmental mutation condition may also include other conditions, which are not limited in the embodiments of the present application. The target image frame may represent scenes such as vehicles entering roundabouts, ramps, intersections, etc. in the second image frame. The target image frame may also represent scenes such as construction on the road or accidents on the road in the second image frame. The target image frame may also represent scenes in the second image frame including changes in the weather of the environment outside the target object.
[0047] The method of the embodiment of the present application effectively protects the sensitive objects in the multimedia data from environmental mutations and prevents information leakage of sensitive objects. The present application optimizes the data desensitization algorithm and processing flow, improves the vehicle-side processing capability, reduces the pressure on the cloud, can quickly respond to environmental changes, and ensures real-time processing of sensitive information. Through image frame recognition and random sampling, computing resources are reasonably allocated, and the data desensitization processing efficiency of multimedia data is improved.
[0048] In some embodiments, the processing flow of the multimedia data processing method is shown as follows: Figure 3 ,like Figure 3 As shown, performing image recognition on multiple image frames to obtain a target image frame in step S102 may include:
[0049] Step S301, extracting features from an image frame to obtain image features in the image frame.
[0050] Step S302, obtaining the set reference features.
[0051] In this embodiment, the image features may include: image features such as color features, texture features, shape features, and edge features in the image frame. The image features may also include other visual features, which are not limited in the embodiment of the present application. The image features may be used to represent the characteristics of the image content in the image frame. The reference features may characterize the image features corresponding to the special entities in the image frame.
[0052] Step S303: determine the similarity between the image feature in the image frame and the reference feature.
[0053] Step S304 : in response to the similarity being greater than a set similarity threshold, determining the corresponding image frame as a target image frame.
[0054] In this embodiment, the similarity may include Euclidean distance or cosine similarity. The similarity threshold may include: a preset similarity.
[0055] As an example, the vehicle-mounted device performs feature extraction on each image frame in the video stream, and extracts the image feature of the license plate number of the vehicle included in the image frame. The reference feature is obtained from the reference feature database. The reference feature may include the image feature of a known special entity (such as the license plate number of a fire truck). The similarity between the image feature in the image frame and the reference feature is determined, and the similarity between the image feature in each image frame and the reference feature is calculated. In response to the similarity being greater than the set similarity of 90%, the corresponding image frame is determined as the target image frame.
[0056] The method of the embodiment of the present application effectively protects special objects included in sensitive objects in multimedia data and prevents information leakage of sensitive objects. The present application optimizes the data desensitization algorithm and processing flow, improves the vehicle-side processing capability, reduces the pressure on the cloud, can quickly respond to environmental changes, and ensures real-time processing of sensitive information. Through image frame recognition and random sampling, computing resources are reasonably allocated, and the data desensitization processing efficiency of multimedia data is improved.
[0057] In some embodiments, the processing flow of the multimedia data processing method is shown as follows: Figure 4 ,like Figure 4 As shown, determining the data processing priority of each image frame based on the target image frame in step S103 may include:
[0058] Step S401 : determining a time range corresponding to a target image frame based on a display time of the target image frame and an attribute feature of the target image frame.
[0059] In this embodiment, the time range may include the display time of the target image frame. The starting time of the time range may be before the display time of the target image frame, and the ending time of the time range may be after the display time of the target image frame; the starting time of the time range may be the display time of the target image frame, and the ending time of the time range may be after the display time of the target image frame; the starting time of the time range may be before the display time of the target image frame, and the ending time of the time range may be the display time of the target image frame. The attribute features of the target image frame may include: the attribute features of the sudden environmental changes or special entities contained in the target image frame. Recognizable features, such as color, shape, texture, or attributes of a specific object
[0060] Step S402: determining the data processing priority of the image frames within the time range as the first priority.
[0061] Step S403: determining the data processing priority of the image frames outside the time range as the second priority.
[0062] In this embodiment, the first priority may be higher than the second priority. The image frames corresponding to the first priority are allocated more resources or attention than the images corresponding to the second priority.
[0063] As an example, the camera on the vehicle acquires a continuous video stream and transmits it to the vehicle-mounted device. The vehicle-mounted device identifies the target image frame containing the traffic accident from the video stream. The display time of the target image frame is determined to be 10:00, and the target image frame is analyzed to obtain the attribute characteristics of the traffic accident contained therein, such as the type of accident, the number of vehicles involved, etc. Based on the display time and attribute characteristics of the target image frame, the time range is determined to be 9:58 to 10:08. Among them, the starting time of the time range, 9:58, represents the display time when the image frame containing the traffic accident first appears in the video stream. The ending time of the time range, 10:08, represents the display time when the image frame containing the traffic accident appears in the video stream for the last time. All image frames within the time range are marked as the first priority. Image frames outside the time range are marked as the second priority.
[0064] The method of the embodiment of the present application effectively protects sensitive objects in multimedia data and prevents information leakage of sensitive objects. The present application optimizes the data desensitization algorithm and processing flow, performs data desensitization of image frames according to different priorities, improves vehicle-side processing capabilities, reduces pressure on the cloud, can quickly respond to environmental changes, and ensures real-time processing of sensitive information. Through image frame recognition and random sampling, computing resources are reasonably allocated, and the data desensitization processing efficiency of multimedia data is improved.
[0065] In some embodiments, the processing flow of the multimedia data processing method is shown as follows: Figure 5 ,like Figure 5 As shown, in step S104, based on the data processing priority of each image frame, data desensitization is performed on the multimedia data to obtain desensitized multimedia data, which may include:
[0066] Step S501: Perform target detection on the image frame of the first priority to obtain sensitive objects included in the image frame of the first priority.
[0067] Step S502: Determine the pixel coordinates of the sensitive object included in the image frame of the first priority.
[0068] Step S503: mosaic processing is performed on the image frame of the first priority based on the pixel coordinates to obtain desensitized multimedia data.
[0069] In this embodiment, target detection can be used in the process of identifying and locating sensitive objects in an image frame. Sensitive objects may include: objects contained in an image frame that require privacy or security protection. Sensitive objects may include: sudden changes in the environment and special entities. Sensitive objects may specifically include: roundabouts, ramps, intersections, construction on the road or accidents on the road, scenes where the weather outside the target object changes, management areas, or fire trucks. Mosaic processing can be used to blur areas in an image frame according to pixel coordinates to cover sensitive objects.
[0070] As an example, the image frames of the first priority include 5 image frames. The vehicle-mounted device performs target detection on the 5 image frames of the first priority, and obtains that the sensitive object included in these 5 image frames is a fire truck. The pixel coordinates of the fire truck included in each image frame are determined. The area where the fire truck is located in each image frame is mosaic processed according to the pixel coordinates to cover the fire truck image included in the 5 image frames, and the mosaic processed image frames are determined as desensitized multimedia data.
[0071] The method of the embodiment of the present application effectively protects sensitive objects in multimedia data and prevents information leakage of sensitive objects. The present application optimizes the data desensitization algorithm and processing flow, performs data desensitization of image frames according to different priorities, improves vehicle-side processing capabilities, reduces pressure on the cloud, can quickly respond to environmental changes, and ensures real-time processing of sensitive information. Through image frame recognition and random sampling, computing resources are reasonably allocated, and the data desensitization processing efficiency of multimedia data is improved.
[0072] In some embodiments, the processing flow of the multimedia data processing method is shown as follows: Figure 6 ,like Figure 6 As shown, in step S104, based on the data processing priority of each image frame, data desensitization is performed on the multimedia data to obtain desensitized multimedia data, which may include:
[0073] Step S601: Obtain the frame rate corresponding to the image frame of the second priority in the multimedia data.
[0074] Step S602: determining a sampling frequency based on a frame rate corresponding to an image frame of a second priority.
[0075] Step S603: extract at least one third image frame from the image frames of the second priority based on the sampling frequency.
[0076] Step S604, in response to the third image frame satisfying the set data desensitization condition, data desensitization is performed on the image frame of the second priority based on the third image frame to obtain desensitized multimedia data.
[0077] In this embodiment, the frame rate may include: the number of frames displayed per second in the multimedia data. The sampling frequency may include: the frequency of extracting samples at a certain time interval in multiple second priority image frames. Meeting the set data desensitization conditions may include: the third image frame is a pre-set image frame that needs to be desensitized, the third image frame includes a sensitive object, and the third image frame is an image frame corresponding to the selection instruction. The selection instruction may be a selection instruction input by the user for a certain image frame of the multimedia data. The data desensitization conditions may also include other conditions, which are not limited by the embodiments of the present application. Sensitive objects may include: objects that need to protect privacy or security contained in the image frame. Sensitive objects may include: sudden environmental changes and special entities. Sensitive objects may specifically include: roundabouts, ramps, intersections, construction on the road or accidents on the road, scenes where the weather outside the target object changes, management areas or fire trucks.
[0078] As an example, the image frames of the second priority include 300 image frames. The frame rate corresponding to the image frames of the second priority in the multimedia data obtained by the vehicle-mounted device is 30 frames. Based on the frame rate corresponding to the image frames of the second priority, the sampling frequency is determined. For example, one image frame is extracted every 5 seconds, that is, 1 frame is extracted every 150 frames, and the sampling frequency is 0.067FPS. Based on the sampling frequency, 2 third image frames are randomly extracted from the 300 image frames. It is determined whether the two third image frames meet the data desensitization conditions, and if one third image frame does not meet the data desensitization conditions, then the third image frame does not need to be subjected to data desensitization processing.
[0079] If another third image frame meets the data desensitization condition, the display time of the third image frame is determined, and the third image frame is analyzed to obtain the attribute characteristics of the contained third image frame. Based on the display time and attribute characteristics of the third image frame, a time range is determined. Among them, the starting time of the time range is before the display time of the third image frame. The end time of the time range is after the display time of the third image frame. All image frames within the time range are marked as first priority. Target detection is performed on the image frames of the first priority to obtain that the image frames of the first priority include sensitive objects. Determine the pixel coordinates of each image frame including the sensitive object. Mosaic processing is performed on the area where the sensitive object in each image frame is located according to the pixel coordinates to cover the sensitive object included in the image frame of the first priority, and the mosaic-processed image frame is determined as desensitized multimedia data.
[0080] The method of the embodiment of the present application effectively protects sensitive objects in multimedia data and prevents information leakage of sensitive objects. The present application optimizes the data desensitization algorithm and processing flow, performs data desensitization of image frames according to different priorities, improves vehicle-side processing capabilities, reduces pressure on the cloud, can quickly respond to environmental changes, and ensures real-time processing of sensitive information. Through image frame recognition and random sampling, computing resources are reasonably allocated, and the data desensitization processing efficiency of multimedia data is improved.
[0081] In some embodiments, the processing flow of the multimedia data processing method is shown as follows: Figure 7 ,like Figure 7 As shown, the multimedia data processing method may further include:
[0082] Step S701: Acquire the data transmission mode of multimedia data.
[0083] Step S702: determining a corresponding multimedia encoding rule based on the data transmission mode.
[0084] Step S703: Encode the image frames in the desensitized multimedia data based on the determined multimedia encoding rule to obtain streaming multimedia data.
[0085] Step S704: based on the data transmission mode, the streaming multimedia data is transmitted to the receiving end.
[0086] In this embodiment, the data transmission mode may include: wired transmission and wireless transmission. Among them, wireless transmission may include: Ethernet, Wi-Fi, 4G / 5G, live streaming media, etc. Multimedia encoding rules may include: a protocol for converting multimedia data into a specific format for transmission and storage, and the specific format may include formats such as H.264, JPEG, and MP3. The specific format may also include the format of other multimedia data, which is not limited by the embodiment of the present application. Encoding the image frames in the desensitized multimedia data may include: encoding a plurality of desensitized image frames according to the selected encoding rules, and splicing them to obtain streaming multimedia data. Streaming multimedia data may include: a multimedia data stream that can be transmitted over the network after encoding and encapsulation. The receiving end may include: a receiving end such as a terminal device, a cloud, and a digital media player, and the receiving end may also include other receiving ends, which are not limited by the embodiment of the present application.
[0087] The method of the embodiment of the present application effectively protects sensitive objects in multimedia data and prevents information leakage of sensitive objects. The present application optimizes the data desensitization algorithm and processing flow, performs data desensitization of image frames according to different priorities, improves the vehicle-side processing capability, and can encode the processed multimedia data before transmitting the data according to the data transmission type of the receiving end, thereby reducing the pressure on the cloud, being able to respond quickly to environmental changes, and ensuring real-time processing of sensitive information. Through image frame recognition and random sampling, computing resources are reasonably allocated, and the data desensitization processing efficiency of multimedia data is improved.
[0088] The following is a description of an exemplary structure of software modules included in the multimedia data processing device 80 provided in the embodiment of the present application. In some embodiments, for example, Figure 8 As shown, the multimedia data processing device 80 may include: an acquisition module 801, used to acquire multiple image frames in multimedia data, where the multimedia data is obtained by a data acquisition device on the target object performing data acquisition on the outside of the target object; an identification module 802, used to perform image recognition on multiple image frames to obtain a target image frame; the target image frame indicates that there is an abnormality outside the target object; a determination module 803, used to determine the data processing priority of each image frame based on the target image frame; and a processing module 804, used to perform data desensitization on the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data.
[0089] In some embodiments, the identification module 802 can be used to: obtain a first image frame and a second image frame from multiple image frames; the display time of the second image frame is after the first image frame; based on the first image frame and the second image frame, determine a change area in the second image frame; the change area characterizes the change between the first image frame and the second image frame; in response to the change area satisfying a set environmental mutation condition, determine the second image frame as a target image frame.
[0090] In some embodiments, the recognition module 802 can be used to: perform feature extraction on the image frame to obtain image features in the image frame; obtain set reference features; the reference features represent image features corresponding to special entities in the image frame; determine the similarity between the image features in the image frame and the reference features; in response to the similarity being greater than a set similarity threshold, determine the corresponding image frame as a target image frame.
[0091] In some embodiments, the determination module 803 can be used to: determine the time range corresponding to the target image frame based on the display time of the target image frame and the attribute characteristics of the target image frame; the time range at least includes the display time of the target image frame; determine the data processing priority of the image frames within the time range as the first priority; determine the data processing priority of the image frames outside the time range as the second priority; the first priority is higher than the second priority.
[0092] In some embodiments, the processing module 804 can be used to: perform target detection on the first-priority image frames to obtain sensitive objects included in the first-priority image frames; determine the pixel coordinates of the sensitive objects included in the first-priority image frames; and perform mosaic processing on the first-priority image frames based on the pixel coordinates to obtain desensitized multimedia data.
[0093] In some embodiments, the processing module 804 can be used to: obtain the frame rate corresponding to the image frame of the second priority in the multimedia data; determine the sampling frequency based on the frame rate corresponding to the image frame of the second priority; extract at least one third image frame from the image frame of the second priority based on the sampling frequency; in response to the third image frame satisfying the set data desensitization condition, perform data desensitization on the image frame of the second priority based on the third image frame to obtain desensitized multimedia data.
[0094] In some embodiments, the multimedia data processing device 80 also includes a transmission module, which can be used to: obtain a data transmission method for multimedia data; determine corresponding multimedia encoding rules based on the data transmission method; encode image frames in the desensitized multimedia data based on the determined multimedia encoding rules to obtain streaming multimedia data; and transmit the streaming multimedia data to a receiving end based on the data transmission method.
[0095] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figures 1 to 7 The present invention can be understood by referring to the description of any one of the accompanying drawings.
[0096] The following is a description of an exemplary structure of a software unit included in the electronic device 90 provided in an embodiment of the present application. In some embodiments, for example, Fig. 9 As shown, the electronic device 90 may be a vehicle-mounted device on a vehicle, and the electronic device 90 may include: a data acquisition unit 901, used to acquire multiple image frames in multimedia data, where the multimedia data is obtained by a data acquisition device on the target object performing data acquisition on the outside of the target object; an image recognition unit 902, used to perform image recognition on multiple image frames to obtain a target image frame; the target image frame indicates that there is an abnormality outside the target object; a priority determination unit 903, used to determine the data processing priority of each image frame based on the target image frame; a data processing unit 904, used to desensitize the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data.
[0097] In some embodiments, the image recognition unit 902 can be used to: obtain a first image frame and a second image frame from multiple image frames; the display time of the second image frame is after the first image frame; based on the first image frame and the second image frame, determine a change area in the second image frame; the change area characterizes the change between the first image frame and the second image frame; in response to the change area satisfying a set environmental mutation condition, determine the second image frame as a target image frame.
[0098] In some embodiments, the image recognition unit 902 can be used to: perform feature extraction on an image frame to obtain image features in the image frame; obtain set reference features; the reference features represent image features corresponding to special entities in the image frame; determine the similarity between the image features in the image frame and the reference features; in response to the similarity being greater than a set similarity threshold, determine the corresponding image frame as a target image frame.
[0099] In some embodiments, the priority determination unit 903 can be used to: determine the time range corresponding to the target image frame based on the display time of the target image frame and the attribute characteristics of the target image frame; the time range at least includes the display time of the target image frame; determine the data processing priority of the image frames within the time range as the first priority; determine the data processing priority of the image frames outside the time range as the second priority; the first priority is higher than the second priority.
[0100] In some embodiments, the data processing unit 904 can be used to: perform target detection on the first-priority image frames to obtain sensitive objects included in the first-priority image frames; determine the pixel coordinates of the sensitive objects included in the first-priority image frames; and perform mosaic processing on the first-priority image frames based on the pixel coordinates to obtain desensitized multimedia data.
[0101] In some embodiments, the data processing unit 904 can be used to: obtain the frame rate corresponding to the image frame of the second priority in the multimedia data; determine the sampling frequency based on the frame rate corresponding to the image frame of the second priority; extract at least one third image frame from the image frame of the second priority based on the sampling frequency; in response to the third image frame satisfying the set data desensitization condition, perform data desensitization on the image frame of the second priority based on the third image frame to obtain desensitized multimedia data.
[0102] In some embodiments, the electronic device 90 also includes a transmission module, which can be used to: obtain a data transmission method for multimedia data; determine corresponding multimedia encoding rules based on the data transmission method; encode image frames in the desensitized multimedia data based on the determined multimedia encoding rules to obtain streaming multimedia data; and transmit the streaming multimedia data to a receiving end based on the data transmission method.
[0103] It should be noted that the description of the electronic device in the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figures 1 to 7 The present invention can be understood by referring to the description of any one of the accompanying drawings.
[0104] According to an embodiment of the present application, the present application also provides a non-transitory computer-readable storage medium.
[0105] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0106] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0107] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0109] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0110] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0111] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0112] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A multimedia data processing method, characterized in that: The method comprises: Acquire multiple image frames in multimedia data, where the multimedia data is obtained by a data acquisition device on the target object acquiring data outside the target object; Performing image recognition on the multiple image frames to obtain a target image frame; the target image frame indicates that an abnormality exists outside the target object; Based on the target image frame, determining the data processing priority of each image frame; The determining of the data processing priority of each image frame based on the target image frame includes: determining a time range corresponding to the target image frame based on the display time of the target image frame and the attribute characteristics of the target image frame; the time range at least includes the display time of the target image frame; determining the data processing priority of the image frames within the time range as a first priority; determining the data processing priority of the image frames outside the time range as a second priority; the first priority is higher than the second priority; Based on the data processing priority of each image frame, desensitizing the multimedia data to obtain desensitized multimedia data; The method of performing data desensitization on the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data includes: obtaining the frame rate corresponding to the image frame of the second priority in the multimedia data; determining a sampling frequency based on the frame rate corresponding to the image frame of the second priority; extracting at least one third image frame from the image frame of the second priority based on the sampling frequency; and performing data desensitization on the image frame of the second priority based on the third image frame in response to the third image frame satisfying a set data desensitization condition to obtain the desensitized multimedia data.
2. The method according to claim 1, characterized in that The performing image recognition on the multiple image frames to obtain a target image frame includes: Obtaining a first image frame and a second image frame from the plurality of image frames; the display time of the second image frame is after the display time of the first image frame; Based on the first image frame and the second image frame, determining a change region in the second image frame; the change region represents a change between the first image frame and the second image frame; In response to the change region satisfying a set environmental mutation condition, the second image frame is determined as the target image frame.
3. The method according to claim 1, characterized in that: The performing image recognition on the multiple image frames to obtain a target image frame includes: Performing feature extraction on the image frame to obtain image features in the image frame; Acquire a set reference feature; the reference feature represents an image feature corresponding to a special entity in the image frame; Determining the similarity between the image feature in the image frame and the reference feature; In response to the similarity being greater than a set similarity threshold, the corresponding image frame is determined as the target image frame.
4. The method according to claim 1, characterized in that: The step of performing data desensitization on the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data includes: Performing target detection on the image frames of the first priority level to obtain sensitive objects included in the image frames of the first priority level; Determine pixel coordinates of the sensitive object in the image frame of the first priority; Mosaic processing is performed on the image frame of the first priority based on the pixel coordinates to obtain the desensitized multimedia data.
5. The method according to claim 1, characterized in that The method further comprises: Acquire a data transmission mode of the multimedia data; Based on the data transmission mode, determining a corresponding multimedia encoding rule; Based on the determined multimedia encoding rule, encoding the image frames in the desensitized multimedia data to obtain streaming multimedia data; Based on the data transmission mode, the streaming multimedia data is transmitted to a receiving end.
6. A multimedia data processing device, characterized in that: The device comprises: An acquisition module, used to acquire multiple image frames in multimedia data, where the multimedia data is acquired by a data acquisition device on the target object through data acquisition outside the target object; A recognition module, configured to perform image recognition on the plurality of image frames to obtain a target image frame; the target image frame indicates that an abnormality exists outside the target object; A determination module, used to determine the data processing priority of each image frame based on the target image frame; The determining of the data processing priority of each image frame based on the target image frame includes: determining a time range corresponding to the target image frame based on the display time of the target image frame and the attribute characteristics of the target image frame; the time range at least includes the display time of the target image frame; determining the data processing priority of the image frames within the time range as a first priority; determining the data processing priority of the image frames outside the time range as a second priority; the first priority is higher than the second priority; A processing module, configured to perform data desensitization on the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data; The method of performing data desensitization on the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data includes: obtaining the frame rate corresponding to the image frame of the second priority in the multimedia data; determining a sampling frequency based on the frame rate corresponding to the image frame of the second priority; extracting at least one third image frame from the image frame of the second priority based on the sampling frequency; and performing data desensitization on the image frame of the second priority based on the third image frame in response to the third image frame satisfying a set data desensitization condition to obtain the desensitized multimedia data.
7. An electronic device, characterized in that: The method according to any one of claims 1 to 5 can be performed, comprising: A data acquisition unit, used to acquire multiple image frames in multimedia data, where the multimedia data is acquired by a data acquisition device on the target object through data acquisition outside the target object; An image recognition unit, configured to perform image recognition on the plurality of image frames to obtain a target image frame; the target image frame indicates that an abnormality exists outside the target object; A priority determination unit, used to determine the data processing priority of each image frame based on the target image frame; The determining of the data processing priority of each image frame based on the target image frame includes: determining a time range corresponding to the target image frame based on the display time of the target image frame and the attribute characteristics of the target image frame; the time range at least includes the display time of the target image frame; determining the data processing priority of the image frames within the time range as a first priority; determining the data processing priority of the image frames outside the time range as a second priority; the first priority is higher than the second priority; A data processing unit, configured to perform data desensitization on the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data; The method of performing data desensitization on the multimedia data based on the data processing priority of each image frame to obtain desensitized multimedia data includes: obtaining the frame rate corresponding to the image frame of the second priority in the multimedia data; determining a sampling frequency based on the frame rate corresponding to the image frame of the second priority; extracting at least one third image frame from the image frame of the second priority based on the sampling frequency; and performing data desensitization on the image frame of the second priority based on the third image frame in response to the third image frame satisfying a set data desensitization condition to obtain the desensitized multimedia data.
8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-5.
