Media content processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a media content processing method and device, electronic equipment and a storage medium, in the process of playing target media content, a face data sequence of a watching user is collected through camera equipment, a face action unit combination sequence is analyzed from the face data sequence, and the target media content is played according to the face action unit combination sequence. And determining an emotional state data sequence of the watching user, and determining a target fragment meeting a set condition from the target media content according to the emotional state data sequence. According to the mode, the target fragment is determined from the target media content based on the collected emotional state of the user, an objective and quantifiable emotional evaluation standard is established, the fragment causing emotional resonance of the user can be accurately captured, multiple types of media contents can be adapted, editing depending on artificial experience is avoided, and the user experience is improved. The efficiency and the quality of fragment extraction and editing are improved, and the cost is reduced at the same time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a media content processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of the Internet, media content such as videos occupies an increasingly important position in people's lives, from daily entertainment to commercial promotion, to knowledge dissemination, the video is ubiquitous, greatly promoting the vigorous development of the video editing market, making it present an extremely active status.

[0003] However, the existing editing technology relies on manual work, has high cost, low efficiency, and strong subjectivity, lacks objective standards, has poor universality, and is difficult to capture segments that can resonate with users. SUMMARY

[0004] The present application provides a media content processing method, device, electronic equipment and storage medium to solve the technical problem that the existing editing technology relies on manual work, has high cost, low efficiency, and strong subjectivity, lacks objective standards, has poor universality, and is difficult to capture segments that can resonate with users.

[0005] In a first aspect, the present application provides a media content processing method, the method comprising:

[0006] In the process of playing the target media content, the face data sequence of the viewing user is collected by the camera device;

[0007] The face action unit combination sequence is parsed from the face data sequence;

[0008] According to the face action unit combination sequence, the emotional state data sequence of the viewing user is determined;

[0009] According to the emotional state data sequence, the target segment that meets the set condition is determined from the target media content.

[0010] In a possible implementation, the face action unit combination sequence is parsed from the face data sequence, comprising:

[0011] The following processing is performed for each face data in the face data sequence to obtain the face action unit combination sequence:

[0012] The action intensity parameter set of the target facial feature region is parsed from the face data;

[0013] According to the preset face action coding system rule, the action intensity parameter set is mapped to the face action unit combination.

[0014] In a possible implementation, the determining of the sequence of emotional state data of the viewing user according to the sequence of facial action unit combinations comprises:

[0015] The following processing is respectively performed for each facial action unit combination in the sequence of facial action unit combinations to obtain the sequence of emotional states of the viewing user:

[0016] Determining an emotional expectation type of a media content segment played when the facial data corresponding to the facial action unit combination is collected;

[0017] According to the emotional expectation type, determining a weight coefficient of each facial action unit in the facial action unit combination;

[0018] According to the weight coefficient of each facial action unit, quantifying the facial action unit combination into emotional state data.

[0019] In a possible implementation, the determining of the target segment meeting the set condition from the target media content according to the sequence of emotional state data comprises:

[0020] Determining a plurality of target emotional state data meeting a local extreme condition from the sequence of emotional state data;

[0021] Determining a target frame corresponding to each of the target emotional state data from the target media content;

[0022] Determining a target segment meeting the set condition from the target media content according to a plurality of the target frames.

[0023] In a possible implementation, the determining of the plurality of target emotional state data meeting the local extreme condition from the sequence of emotional state data comprises:

[0024] Initializing a sliding window with a length of N, and aligning a start end of the sliding window with a first emotional state data of the sequence of emotional state data;

[0025] Obtaining N emotional state data in the current sliding window;

[0026] Marking the emotional state data at the center position of the current sliding window as a first emotional state data;

[0027] In a case where the first emotional state data is greater than other emotional state data in the current sliding window, determining the first emotional state data as a target emotional state data meeting the local extreme condition;

[0028] Sliding the current sliding window by M emotional state data towards the end of the sequence of emotional state data, and returning to perform the steps of obtaining N emotional state data within the current sliding window and thereafter until the end of the sliding window reaches the end of the sequence of emotional state data.

[0029] In a possible implementation, the determining the target segment satisfying the set condition from the target media content according to the target frames comprises:

[0030] Sequentially traversing the target frames, and when the first target frame is traversed, marking the first target frame as a target segment start frame;

[0031] Starting from the second target frame traversed, determining a frame interval between the target frame currently traversed and a previous target frame; in a case where the frame interval is determined to be greater than a set frame interval threshold, marking the target frame currently traversed as a target segment start frame, and marking the previous target frame as a target segment end frame;

[0032] Determining the target segment satisfying the set condition from the target media content based on the adjacent target segment start frame and the target segment end frame.

[0033] In a possible implementation, the method further comprises:

[0034] Performing segment fusion processing based on the target segments determined by the plurality of viewing users to obtain a final target segment.

[0035] In a possible implementation, the performing segment fusion processing based on the target segments determined by the plurality of viewing users to obtain a final target segment comprises:

[0036] Taking an average of start frame positions of the target segments to obtain a fusion start frame, and taking an average of end frame positions of the target segments to obtain a fusion end frame;

[0037] Determining the final target segment from the target media content according to the fusion start frame and the fusion end frame.

[0038] In a second aspect, the present application provides a media content processing apparatus, the apparatus comprising:

[0039] A data acquisition module configured to acquire a sequence of facial data of a viewing user by using a camera device in a process of playing a target media content;

[0040] A data analysis module configured to analyze a sequence of facial action unit combinations from the sequence of facial data;

[0041] an emotion state determining module, configured to determine a sequence of emotion state data of the viewing user according to the sequence of facial action unit combinations;

[0042] a target segment determining module, configured to determine a target segment meeting a set condition from the target media content according to the sequence of emotion state data.

[0043] In a possible implementation, the data analyzing module is specifically configured to:

[0044] the following processing is performed on each piece of facial data in the sequence of facial data respectively, to obtain the sequence of facial action unit combinations:

[0045] an action intensity parameter set of a target facial feature region is analyzed from the facial data;

[0046] the action intensity parameter set is mapped to a facial action unit combination according to a preset facial action coding system rule.

[0047] In a possible implementation, the emotion state determining module is specifically configured to:

[0048] the following processing is performed on each facial action unit combination in the sequence of facial action unit combinations respectively, to obtain the sequence of emotion state data of the viewing user:

[0049] an emotion expectation type of a media content segment played when the facial data corresponding to the facial action unit combination is collected is determined;

[0050] a weight coefficient of each facial action unit in the facial action unit combination is determined according to the emotion expectation type;

[0051] the facial action unit combination is quantified into emotion state data according to the weight coefficient of each facial action unit.

[0052] In a possible implementation, the target segment determining module is configured to:

[0053] an emotion state data determining unit, configured to determine a plurality of target emotion state data meeting a local extreme condition from the sequence of emotion state data;

[0054] a target frame determining unit, configured to determine a target frame corresponding to each of the target emotion state data from the target media content;

[0055] a target segment determining unit, configured to determine a target segment meeting a set condition from the target media content according to a plurality of the target frames.

[0056] In a possible implementation, the emotion state data determining unit is specifically configured to:

[0057] initializing a sliding window with a length of N, and aligning a start of the sliding window with a first affective state data of the sequence of affective state data;

[0058] obtaining N affective state data in a current sliding window;

[0059] marking the affective state data at a center position of the current sliding window as a first affective state data;

[0060] in a case where the first affective state data is greater than other affective state data in the current sliding window, determining the first affective state data as a target affective state data satisfying a local extremum condition;

[0061] sliding the current sliding window by M affective state data along the sequence of affective state data in a direction towards an end, and returning to execute the obtaining N affective state data in a current sliding window and the subsequent steps until an end of the sliding window reaches an end of the sequence of affective state data.

[0062] In a possible implementation, the target segment determining unit is specifically configured to:

[0063] sequentially traversing a plurality of the target frames, and marking a first target frame as a target segment start frame when the first target frame is traversed;

[0064] starting from a second target frame that is traversed, determining a frame interval between a target frame that is currently traversed and a previous target frame of the target frame that is currently traversed, and marking the target frame that is currently traversed as a target segment start frame and marking the previous target frame as a target segment end frame in a case where the frame interval is greater than a set frame interval threshold;

[0065] determining a target segment satisfying a set condition from the target media content based on adjacent target segment start frames and target segment end frames.

[0066] In a possible implementation, the apparatus further includes:

[0067] a fusion processing module configured to perform segment fusion processing based on a plurality of target segments determined by a plurality of the watching users to obtain a final target segment.

[0068] In a possible implementation, the fusion processing module is specifically configured to:

[0069] averaging start frame positions of the plurality of target segments to obtain a fusion start frame, and averaging end frame positions of the plurality of target segments to obtain a fusion end frame;

[0070] According to the fusion start frame and the fusion end frame, a final target segment is determined from the target media content.

[0071] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory, the processor is configured to execute a media content processing program stored in the memory to implement the media content processing generation method in any one of the first aspect.

[0072] In a fourth aspect, the present application provides a storage medium, the storage medium stores one or more programs, the one or more programs can be executed by one or more processors to implement the media content processing method in any one of the first aspect.

[0073] The above technical solutions provided by the embodiments of the present application have the following advantages compared with the prior art: the method provided by the embodiments of the present application, in the process of playing the target media content, the face data sequence of the watching user is collected by the camera device, the face action unit combination sequence is parsed from the face data sequence, the emotional state data sequence of the watching user is determined according to the face action unit combination sequence, and the target segment meeting the set condition is determined from the target media content according to the emotional state data sequence. This way determines the target segment from the target media content based on the collected emotional state of the user, establishes an objective and quantifiable emotional evaluation standard, can accurately capture the segment that causes the user to have emotional resonance, adapts to various types of media content, avoids relying on manual experience editing, improves the efficiency and quality of segment extraction and editing, and reduces the cost. BRIEF DESCRIPTION OF DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows, and obviously, other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0076] One or more embodiments are exemplarily illustrated by the pictures in the drawings corresponding thereto, and these exemplary illustrations do not constitute a limitation on the embodiments, and the elements with the same reference numerals in the drawings represent similar elements, unless otherwise specified, and the drawings do not constitute a proportional limitation.

[0077] Figure 1 An embodiment flowchart of a media content processing method provided by the embodiments of the present application is shown in the following table:

[0078] Figure 2 An embodiment flowchart of another media content processing method provided by an embodiment of the present application is provided.

[0079] Figure 3 An embodiment flowchart of another media content processing method provided by an embodiment of the present application is provided.

[0080] Figure 4 An embodiment flowchart of another media content processing method provided by an embodiment of the present application is provided.

[0081] Figure 5 A block diagram of a media content processing device provided by an embodiment of the present application is provided.

[0082] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0083] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0084] The following disclosure provides many different embodiments, or examples, for implementing different structures of the present application. For the purpose of simplicity and clarity, the description of the specific examples in the following text will be described. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeatedly refer to the numbers and / or letters in different examples. Such repetition is for the purpose of simplification and clarity, and it does not indicate the relationship between the various embodiments and / or settings discussed.

[0085] To solve the technical problems of the existing editing technology, which relies on manual work, has high cost, low efficiency, strong subjectivity, lacks objective standards, poor universality, and is difficult to capture the segments that can resonate with users, the present application provides a media content processing method, device, electronic device and storage medium, determines a target segment from target media content based on the emotional state of the collected user, establishes an objective and quantifiable emotional evaluation standard, can accurately capture the segment that causes the user to resonate emotionally, adapts to various types of media content, avoids relying on manual experience editing, improves the efficiency and quality of segment extraction and editing, and reduces the cost.

[0086] In an embodiment, the execution subject of the embodiments of the present application is an electronic device. The electronic device can be a hardware device or software that supports network connection to provide various network services. When the device is hardware, it can support various electronic devices with display screens, including but not limited to smartphones, tablets, laptop computers, desktop computers, servers, etc. When the device is software, it can be installed in the above-mentioned electronic devices listed.

[0087] Figure 1 An embodiment flowchart of a media content processing method provided by the embodiments of the present application is shown in FIG. 1, including the following steps: Figure 1

[0088] Step 101, in the process of playing target media content, a facial data sequence of a watching user is collected by a camera device.

[0089] Target media content refers to media content to be processed, where the media content can be related content such as films, television series, videos, etc. spread through various media channels.

[0090] In the embodiments of the present application, in order to process the target media content, an electronic device with a display screen is used to play the target media content for the user to watch, and in the process of playing the target media content, a facial data sequence of a watching user is collected by a camera device. Exemplarily, an electronic device such as a smart TV or a tablet computer can be used to play the target media content, and the embodiments of the present application do not limit the playing approach and playing device of the target media content.

[0091] The camera device refers to a tool for capturing dynamic images. In the embodiments of the present application, the camera device can be a camera module integrated on the electronic device playing the target media content, or a camera head in communication connection with the execution subject of the embodiments of the present application. Exemplarily, the camera device in the embodiments of the present application can be a TureDepth camera, which can scan the user's face, capture facial motion information, and then collect facial expressions and motions to obtain facial data of the user.

[0092] Further, the facial data obtained by the TureDepth camera refers to the facial mixed shape data obtained after the TureDepth camera captures the user's facial motion in real time. Exemplarily, the above-mentioned facial mixed shape data can be regarded as a dictionary, which contains a series of key-value pairs, where the key refers to a representative motion of the face, and the value refers to a floating point number between 0 and 1, and the larger the value, the larger the motion amplitude of the face corresponding to the key. For example, the larger the value corresponding to the key of a smiling mouth corner, the larger the amplitude of the user's smile.

[0093] ​In an embodiment, during playing of the target media content, a sequence of facial data of the viewing user is collected by the camera device. The sequence of facial data includes a plurality of facial data arranged in a collection order. For example, during playing of the target media content, a plurality of facial actions of the viewing user are collected by the camera device in real time, and the plurality of facial actions are converted into the sequence of facial data.

[0094] For example, assuming that during playing of a video, facial data 1, facial data 2, facial data 3, and facial data 4 are collected by the camera device in sequence, the sequence of facial data can be [facial data 1, facial data 2, facial data 3, facial data 4].

[0095] Step 102, parsing a sequence of facial action unit combinations from the sequence of facial data.

[0096] A facial action unit is a standardized classification of human facial muscle movements based on FACS (Facial Action Coding System), which corresponds to a quantitative indicator of a specific facial muscle movement and is used to accurately describe the basic constituent unit of facial expressions.

[0097] For example, 52 basic AUs (Action Units) can be applied to perform emotional analysis on facial actions of a user. Specifically, the 52 AUs include AU1-AU52, each of which represents a different facial muscle movement, such as AU1 representing “inner eyebrow raise” and AU12 representing “mouth corner up”, and so on. These facial action units can be used individually or in combination to describe and analyze various facial expressions of a user when watching media content.

[0098] In an embodiment, a corresponding sequence of facial action unit combinations is parsed from the sequence of facial data, which includes action amplitude conditions of each feature point of the user’s face. Embodiments of the present application can extract facial feature point data in the sequence of facial data that can be used to identify the emotional state of the user, map it to a sequence of facial action unit combinations in the facial action coding system, and then determine the emotional state of the user based on the sequence of facial action unit combinations.

[0099] A facial action unit can map mixed shape data of a face collected by a camera to a quantitative indicator of a specific facial muscle movement in a facial coding system. For example, a value of a facial action unit can be a floating-point number of 0.1-1.

[0100] In an embodiment, the specific implementation of resolving the corresponding facial action unit combination sequence from the facial data sequence is: the following processing is performed on each facial data in the facial data sequence respectively to obtain the facial action unit combination sequence: the action intensity parameter set of the target facial feature region is resolved from the facial data; the action intensity parameter set is mapped to the facial action unit combination according to the preset facial action coding system rule.

[0101] The target facial feature region refers to the facial feature region which is focused on in the facial action analysis and emotion calculation process of the embodiments of the present application. Exemplarily, the target facial feature can refer to the eyebrows, eyes, mouth, nose, and cheeks and chin. After the facial mixed shape data of these target facial feature regions is screened from the facial data, data support is provided for subsequent facial action analysis.

[0102] The action intensity parameter set refers to the facial mixed shape data of the target facial feature region contained in the facial data, which covers the representative action of the target facial feature region and can comprehensively capture the facial muscle action related to emotion and accurately reflect the external performance of various emotions of the user, thereby providing rich raw data support for subsequent emotion analysis.

[0103] Exemplarily, the facial mixed shape data conforming to the target facial feature region is first screened from the facial data collected by the camera, and then it is resolved and mapped to the corresponding AU combination in FACS according to the set rule in the facial action coding system, wherein one AU combination includes at least one AU, for example, the [AU1+AU3] combination includes two facial action units AU1 and AU3, wherein AU1 represents "inner eyebrow up", AU3 represents "eyebrow lowerer and closer", the [AU5] combination includes one facial action unit AU5, and represents "levator muscle of upper eyelid movement causes upper eyelid to lift and pull back, thereby making eyes wide open", and the like. The above processing mode is adopted for each facial data in the facial data sequence to obtain the final facial action unit combination sequence.

[0104] For example, assume that in the preset facial action coding system rules, the facial blendshape data "mouthSmileLeft" and "mouthSmileRight" that meet the requirements correspond to the facial action unit combination [AU12], wherein AU12 represents "upturned corners of the mouth". The facial blendshape data "eyeLookDownLeft" corresponds to the facial action unit combination [AU45+AU7], wherein AU45 represents "wink", and AU7 represents "tightening of eyelids". The facial blendshape data "JawOpen" corresponds to the facial action unit combination [AU25+AU26], wherein AU25 represents "lower jaw pulled down", and AU26 represents "jaw open". Then, according to the preset facial action coding system rules in the above example, the mapping processing is performed on each facial data in the facial data sequence, and the facial action unit combination corresponding to each facial data in the facial data sequence is obtained, and then the final facial action unit combination sequence is obtained.

[0105] In step 103, the emotional state data sequence of the watching user is determined according to the facial action unit combination sequence.

[0106] The emotional state data can be used to reflect the emotion or mood of the user when watching a certain video frame, and can also reflect the intensity of the emotion or mood of the user, that is, the quantitative value of the emotion or mood. For example, the emotional state data of the user is [1st frame, happy, 0.9], wherein 1 frame represents that the user watches the 1st frame of the media content, happy represents the emotional state of the user when watching the 1st frame of the media content, and 0.9 represents the intensity of the emotional state of the user when watching the 1st frame of the media content. The emotional state data 1 indicates that the user feels happy when watching the 1st frame of the target media content, and the intensity of this emotional state is very high, indicating that the user is very happy. Similarly, the emotional state data of the user is [5th frame, fear, 0.7], indicating that the user feels fear when watching the 5th frame of the target media content, and the intensity of this emotional state is 0.7.

[0107] The facial action unit combination can reflect the muscle movement of the facial feature region of the user, and can also reflect the movement amplitude. Then, the emotional state of the user can be analyzed according to the muscle movement of the facial feature region of the user and the movement amplitude. For example, the mapping relationship between the emotional state data and the facial action unit combination can also be preset in advance, so as to determine the emotional state of the user according to the obtained facial action unit combination of the user.

[0108] In an embodiment, an exemplary implementation of determining a sequence of emotional state data of a watching user according to a sequence of facial action unit combinations includes: for each facial action unit combination in the sequence of facial action unit combinations, finding emotional state data corresponding to the facial action unit combination from a pre-set mapping relationship between emotional state data and facial action unit combinations, and then acquiring a time point corresponding to the facial action unit combination, and determining the found emotional state data and a corresponding video frame as emotional state data of the user in the process of watching the target media content. Exemplarily, the pre-set emotional state data includes emotional / feeling state and emotional intensity value.

[0109] For example, assuming that the sequence of facial action unit combinations is [facial action unit combination 1, facial action unit combination 2], the acquired facial action unit combination 1 is [AU6+AU12] combination, the corresponding emotional state data of the [AU6+AU12] combination is found to be [happy, 0.9] from the pre-set mapping relationship between emotional state data and facial action unit combinations, the video frame corresponding to the facial action unit combination is the first frame, the emotional state data is [the first frame, happy, 0.9], and [the first frame, happy, 0.9] is determined as emotional state data 1 of the watching user. Assuming that the acquired facial action unit combination 2 is [AU4+AU15] combination, the corresponding emotional state data of the [AU4+AU15] combination is found to be [sad, 0.75] from the pre-set emotional state data, the video frame corresponding to the facial action unit combination is the seventh frame, and [the seventh frame, sad, 0.75] is determined as emotional state data 2 of the watching user. Finally, the sequence of emotional state corresponding to the above sequence of facial action unit combinations is {[the first frame, happy, 0.9], [the seventh frame, sad, 0.75]}. This is only an example, and how to obtain the sequence of emotional state will be described in the following embodiments, which are not limited in the embodiments.

[0110] Step 104, determining a target segment satisfying a set condition from the target media content according to the sequence of emotional state data.

[0111] The set condition is a rule or condition used for screening the target media content, and its essence is to define a target segment meeting a content value judgment standard in a specific scene from the target content through a quantitative or logical standard. Exemplarily, the set condition is used to define a most intense conflict scene as the target segment from the target media content.

[0112] In an embodiment, an exemplary implementation of determining a target segment satisfying a set condition from the target media content according to the sequence of emotional state data includes: first, parsing the sequence of emotional state data to obtain a plurality of emotional state data, screening out the emotional state data whose emotional state or emotional intensity value satisfies a first set condition, then according to the screened sequence of emotional state data, obtaining the video frames in each of the emotional state data, screening out the video frames satisfying a second set condition, combining into a corresponding sequence of video frames, and finally screening out the target segment from the target media content according to the sequence of video frames.

[0113] The first set condition can be a screening rule of the preset emotional intensity value or a screening rule of the emotional state. Exemplarily, the first set condition can be a preset fixed threshold value for screening the emotional state data corresponding to the emotional intensity value satisfying the threshold condition, or a preset emotional state for screening the emotional state data consistent with the preset emotional state.

[0114] The second set condition is a rule for screening the video frames from the emotional state data, which is essentially used for screening the video frame information satisfying the merging threshold condition from the emotional state data, and then determining the target segment according to the acquisition time information. Exemplarily, the second set condition can be a preset fixed threshold value for screening the video frames satisfying the merging threshold from the emotional state data, and merging to obtain the final target segment.

[0115] For example, if the sequence of emotional state data is {[1st frame, happy, 0.9], [7th frame, sad, 0.75], [8th frame, fear, 0.83]}, the emotional state data whose emotional intensity value is greater than or equal to the set first threshold value 0.5 is screened out: {[1st frame, happy, 0.9], [7th frame, sad, 0.75], [8th frame, fear, 0.83]}, and then the corresponding sequence of video frames [1st frame, 7th frame, 8th frame] is obtained, the interval between any two video frames in the sequence of video frames is calculated: 6 frames, 1 frame, wherein 1 frame is less than the preset second threshold value 2 frames, so the two video frames corresponding to 1 frame [7th frame, 8th frame] are screened out, and finally the content segment from the 7th frame to the 8th frame in the target media content is determined as the target segment.

[0116] In addition, the target segment can also be screened out from the target media content in other ways, for example, the emotional state data corresponding to the emotional state (e.g., happy) satisfying the set condition in the obtained sequence of emotional state can be screened out, and the final target segment is determined from the target media content according to the screened emotional state data (e.g., the emotional state data containing the emotional state of happy), which is not limited in the embodiments of the present application.

[0117] The method provided in the embodiments of the present application can determine the target segment from the target media content based on the collected emotional state of the user, establish objective and quantifiable emotional evaluation criteria, accurately capture the segment that causes the user to have emotional resonance, adapt to various types of media content, avoid relying on manual experience editing, improve the efficiency and quality of segment extraction and editing, and reduce costs.

[0118] Figure 2 The embodiment flowchart of another media content processing method provided in the embodiments of the present application is based on the flowchart shown in Figure 1 and mainly describes how to determine the emotional state data sequence of the viewing user, as shown in Figure 2 and includes the following steps:

[0119] Step 201: In the process of playing the target media content, facial data sequence of the viewing user is collected through a camera device.

[0120] Step 202: Facial action unit combination sequence is parsed from the facial data sequence.

[0121] For steps 201-202, refer to the detailed description of the related embodiments described above. Figure 1

[0122] Step 203: The following processing is performed for each facial action unit combination in the facial action unit combination sequence to obtain the emotional state sequence of the viewing user: determining the emotional expectation type of the media content segment played when the facial data corresponding to the facial action unit combination is collected, determining the weight coefficient of each facial action unit in the facial action unit combination according to the emotional expectation type, and quantifying the facial action unit combination into emotional state data according to the weight coefficient of each facial action unit.

[0123] The emotional expectation type refers to the emotional response type of the audience that matches the content style or positioning of the media content segment played when the facial data corresponding to the facial action unit combination is collected.

[0124] The emotional expectation type can be obtained through a preset emotional analysis model. The media content segment played when the facial data corresponding to the facial action unit combination is collected is input into the emotional analysis model to obtain the emotional expectation type corresponding to the segment. The final emotional expectation type can also be determined through other ways, which are not limited in the embodiments of the present application. ​

[0125] In an embodiment, the following processing is performed respectively for each group of facial action unit combinations in the facial action unit combination sequence to obtain the sequence of emotional states of the viewing user: determining an emotional expectation type of a media content segment played when the facial data corresponding to the facial action unit combination is collected. According to the emotional expectation type, the specific implementation manner of the weight coefficient of each facial action unit in the facial action unit combination is: searching for an emotional state corresponding to the current facial action unit combination from a preset emotional state model, wherein the emotional state model includes a corresponding relationship between facial action unit combinations and emotional states. Determining a play time point when the facial data corresponding to the facial action unit combination is collected as video frame information. According to a preset emotional model, determining a target facial action unit corresponding to the emotional expectation type from the facial action unit combination, wherein the emotional model includes a corresponding relationship between emotional expectation types and facial action units; determining the weight coefficient of the target facial action unit according to a first weight setting rule, and determining the weight coefficients of other facial action units in the facial action unit combination except the target facial action unit according to a second weight setting rule; wherein the weight coefficient of the target facial action unit is greater than the weight coefficients of the other facial action units.

[0126] In the above embodiment, the greater the weight coefficient corresponding to the facial action unit, the stronger the correlation between the facial action unit and the emotional expectation type, and vice versa.

[0127] For example, different facial action unit combinations corresponding to different emotional expectation types are different, and the key facial action units relied on are also different. Therefore, the influence of the key facial action units needs to be highlighted by dynamically setting the weight coefficients in combination with the emotional expectation type. Based on the emotional expectation type determined in the above steps and the preset emotional model, a target facial action unit corresponding to the emotional expectation type is determined from the collected facial action unit combination of the user. Then, a higher weight coefficient is assigned to the target action unit to increase the influence of the target facial action unit in the emotional quantization process, and conversely, the weight coefficients of the facial action units other than the target facial action unit are reduced to reduce their influence in the emotional quantization process.

[0128] The first weight setting rule is a setting rule for adjusting the weight coefficient to be greater than a weight threshold. The second weight setting rule is a setting rule for adjusting the weight coefficient to be less than the weight threshold. The specific manner of adjusting the weight coefficient is not limited in the embodiments of the present application.

[0129] For example, if the emotional expectation type is "happy", the corresponding target facial action units are determined according to the preset emotional model "happy", and it is assumed that the target facial action units include AU12 ("upturned corners of the mouth") and AU6 ("eye narrowing"), then the weight coefficients of AU12 and AU6 are set to be higher than the weight threshold 0.5 according to the first weight setting rule, and the weight coefficients of other facial action units, such as AU1 ("inner eyebrow raising"), are lower than the weight threshold 0.5.

[0130] In addition, in actual applications, the weight coefficients can also be dynamically adjusted by learning user feedback, content effect data, and the like through big data, so as to improve the reliability of the weight coefficients.

[0131] The facial action unit is a quantitative index corresponding to a specific facial muscle movement, and its value is a floating point number ranging from 0.1 to 1.

[0132] In an embodiment, according to the weight coefficients of the facial action units, an exemplary implementation of combining and quantifying the facial action units into the emotional state data includes: weighting and summing each facial action unit and its corresponding weight coefficient, determining the weighted sum result as an emotional intensity value corresponding to the emotional state, and combining the emotional state, the video frame information, and the emotional intensity value to obtain the final emotional state data after the facial action units are combined and quantified.

[0133] For example, if the emotional expectation type of the target segment is "happy", the facial action unit combination is AU12=0.8, the weight coefficient is 0.8, AU6=0.3, the weight coefficient is 0.6, and AU1=0.1, the weight coefficient is 0.5, then the calculated emotional intensity value is (0.8x0.8)+(0.3x0.6)+(0.1x0.5)=0.322, the playing time point of the media content when the facial data corresponding to the facial action unit combination is collected is the first frame, the emotional state corresponding to the facial action unit combination is happy, and the emotional state data corresponding to the facial action unit combination of the user collected by the camera is [1 frame, happy, 0.322].

[0134] For a plurality of facial action unit combinations in the facial action unit combination sequence, the plurality of emotional state data, i.e., the final emotional state data sequence, are obtained by using the above-mentioned manner.

[0135] In step 204, the target segment satisfying the set condition is determined from the target media content according to the emotional state data sequence.

[0136] For step 204, refer to the description of other related embodiments.

[0137] By Figure 2The related description of the embodiment shown directly associates the emotion expectation type with the type of the media content segment, dynamically adjusts the weight threshold value by using the emotion expectation type, obtains a more accurate and more scene-adaptive emotion state sequence, can reflect the most real emotion state of the user, and finally provides reliable data support for fine processing and application of the media content.

[0138] Figure 3 Another embodiment flowchart of the media content processing method provided by the embodiment of the present application is shown in Figure 1 Based on the flow shown, the present embodiment mainly describes how to determine a target segment meeting a set condition from the target media content, for example, as shown in Figure 3 As shown, the method comprises the following steps:

[0139] Step 301, in the process of playing the target media content, the facial data sequence of the watching user is collected by the camera device.

[0140] Step 302, the facial action unit combination sequence is parsed from the facial data sequence.

[0141] Step 303, the emotion state data sequence of the watching user is determined according to the facial action unit combination sequence.

[0142] For steps 301-303, refer to the description of the related embodiments described above.

[0143] Step 304, a plurality of target emotion state data meeting the local extreme condition is determined from the emotion state data sequence.

[0144] Step 305, the target frame corresponding to each target emotion state data is determined from the target media content.

[0145] For steps 304-305, the following is a unified description:

[0146] The local extreme condition can refer to a rule or condition meeting the local maximum. For example, the local maximum condition can be the maximum value in a certain part of the value.

[0147] In an embodiment, the exemplary implementation of determining a plurality of target emotional state data satisfying the local extreme condition from the sequence of emotional state data comprises: initializing a sliding window with a length of N, and aligning the start of the sliding window with the first emotional state data of the sequence of emotional state data; obtaining N emotional state data in the current sliding window; marking the emotional state data at the center of the current sliding window as the first emotional state data; in the case that the first emotional state data is greater than other emotional state data in the current sliding window, determining the first emotional state data as the target emotional state data satisfying the local extreme condition; sliding the current sliding window by M emotional state data towards the end of the sequence of emotional state data, and returning to execute the steps of obtaining N emotional state data in the current sliding window and the following steps until the end of the sliding window reaches the end of the sequence of emotional state data.

[0148] The sliding window can refer to a fixed-length window. For example, the fixed length of the sliding window can be 15 frames, 25 frames, or 8 frames, and the sliding window can be moved frame by frame from the start frame to the end of the sequence of emotional state data. The embodiments of the present application do not limit this. In addition, in this embodiment, the value of the emotional state data is the corresponding emotional intensity value.

[0149] The embodiments of the present application use a sliding window to perform calculation, which overcomes the problem that the traditional threshold method is easy to misjudge the instantaneous abnormal fluctuation of the viewing user in watching the video (for example, the short-term facial jitter caused by sudden light change) as the target emotional state data satisfying the local extreme condition. The sliding window mechanism can ensure that the detected emotional state data is a sustained expression, rather than accidental noise data, that is, the misjudgment of transient expression change. In addition, only when the emotional state data at the center of the sliding window is the maximum value, the target emotional state data can be determined, which can avoid misjudging the “rising phase” or “falling phase” of the user's emotion as the maximum value of the emotional state data, accurately find the core frame of the emotional expression, identify the maximum value of the sustained emotional state data, and help to determine the continuous highlight segment in line with the viewing habit of the audience, rather than the scattered instantaneous frame segment.

[0150] For example, if the fixed length of the sliding window is 15 frames, the video frame information in the emotional state data is taken as the horizontal axis, and the emotional intensity value is taken as the vertical axis to construct a rectangular sliding window, and the emotional state data curve is constructed in the window according to the emotional state data sequence. First, the starting end of the sliding window is aligned with the first emotional state data in the emotional state data sequence, at this time, 15 emotional state data in the sliding window are obtained, and the emotional state data at the center point position of the horizontal axis in the sliding window of 15 frame length is obtained, that is, the emotional state data at the 8th frame length position in the window is obtained, and in the case that the emotional state data is greater than the other 14 emotional state data, the emotional state data is determined as the target emotional state data. The window is slid by 1 frame length from front to back, and another 15 emotional state data are obtained, and at this time, the first emotional state data is the second emotional state data in the emotional state data sequence, the same processing as above is performed, and until the last emotional state data in the emotional state data sequence is obtained in the sliding window, the sliding window is stopped, and finally a plurality of target emotional state data, that is, the target emotional state data satisfying the local extreme condition, are obtained.

[0151] For example, assuming that the fixed length of the sliding window is 3 frames, and the emotional state data sequence is {[1st frame, happy, 0.7}, {2nd frame, happy, 0.9}, {3rd frame, happy, 0.6}, {4th frame, happy, 0.7}, {5th frame, happy, 0.8}, {6th frame, happy, 0.65}, {7th frame, happy, 0.9}, {8th frame, happy, 0.5}}, the emotional state curve is constructed according to the above emotional state data sequence, first, 3 emotional state data are obtained from the sliding window: [1st frame, happy, 0.7], [2nd frame, happy, 0.9], [3rd frame, happy, 0.6], and the emotional state data at the center point of the horizontal axis in the sliding window, that is, the emotional state data at the 3rd frame position: [2nd frame, happy, 0.9], which is greater than the other two emotional state data (0.7, 0.6), satisfies the local extreme condition, and [2nd frame, happy, 0.9] is determined as the target emotional state data. The window is slid by 1 frame length from front to back, and the above processing is repeated, and finally a plurality of target emotional state data are obtained: [2nd frame, happy, 0.9], [5th frame, happy, 0.8], [7th frame, happy, 0.9].

[0152] In an embodiment, an example implementation of determining the target frame corresponding to each target emotional state data from the target media content includes: based on the target emotional state data determined in the above step 304, the video frame information corresponding thereto is obtained, and the video frame information corresponding to the target emotional state data is determined as the target frame.

[0153] For example, based on the multiple target emotional state data obtained in the above steps: [frame 2, happy, 0.9], [frame 5, happy, 0.8], [frame 7, happy, 0.9], the video frame information in which is obtained: frame 2, frame 5, frame 7, and the video frame information is determined as the target frame.

[0154] Step 306, determining a target segment satisfying the set condition from the target media content according to the multiple target frames.

[0155] In an embodiment, an exemplary implementation of determining a target segment satisfying the set condition from the target media content according to the multiple target frames includes: sequentially traversing the multiple target frames, when the first target frame is traversed, marking the first target frame as a target segment start frame; starting from the second target frame traversed, determining the frame interval between the target frame currently traversed and the previous target frame; in the case that the frame interval is greater than the set frame interval threshold, marking the target frame currently traversed as a target segment start frame, and marking the previous target frame as a target segment end frame; in the case that the frame interval is less than or equal to the set frame interval threshold, marking the target frame currently traversed as a target segment end frame. Based on the adjacent target segment start frame and the target segment end frame, the target segment satisfying the set condition is determined from the target media content.

[0156] For example, the target frames include frame 2, frame 5, frame 7, sequentially traversing the multiple target frames, marking the first target frame traversed: frame 2 as a target segment start frame, traversing the second target frame: frame 5, calculating the interval = 5-2 = 3, 3 is greater than the set frame interval threshold 2, marking the frame 5 as a target segment start frame, and marking the frame 2 as a target segment end frame, at this time the frame 2 is both a target segment start frame and a target segment end frame, so the target frame is discarded. Then traversing the third target frame, calculating the interval = 7-5 = 2, 2 is less than the set frame interval, marking the target frame 7 currently traversed as a target segment end frame. At this time, the second target frame: frame 5 is a target segment start frame, and the third target frame: frame 7 is a target segment end frame, and the media content segment from frame 5 to frame 7 in the target media content segment is determined as a target segment.

[0157] As another possible implementation, based on the adjacent target segment start frame and the target segment end frame, multiple target segments satisfying the set condition can be determined from the target media content.

[0158] For example, in the process of sequentially traversing the target frames and marking the target segment start frame and the target segment end frame, multiple groups of adjacent target segment start frames and target segment end frames can be finally obtained, each group of adjacent target segment start frames and target segment end frames corresponds to a target segment, and multiple target segments can be finally obtained.

[0159] By Figure 3 According to the related description of the embodiments shown, the target emotional state data is determined by using the sliding window mechanism, which can avoid noise interference, misjudge the accidental emotional fluctuations or the "rising phase" of emotional expression or the "falling phase" of emotional expression of the user as the target emotional state data, ensure the continuity of the target emotional state data, and further ensure the accuracy of the target segment obtained, so that the target segment obtained can cause the emotional resonance of the user, is applied to the editing technology, avoids relying on artificial experience, greatly improves the efficiency and quality of the segment extraction and editing, and reduces the cost.

[0160] Figure 4 The embodiment flowchart of another media content processing method provided by the embodiments of the present application is based on the flowchart shown in Figure 1 Based on the flow shown, the following steps are mainly described for how to obtain the final target segment based on the multiple target segments determined based on the emotional state data sequences of multiple users.

[0161] Step 401, in the process of playing the target media content, the facial data sequence of the watching user is collected by the camera device.

[0162] Step 402, the facial action unit combination sequence is parsed from the facial data sequence.

[0163] Step 403, the emotional state data sequence of the watching user is determined according to the facial action unit combination sequence.

[0164] Step 404, the target segment meeting the set condition is determined from the target media content according to the emotional state data sequence.

[0165] For steps 401-404, refer to the description of the related embodiments described above.

[0166] Step 405, based on the multiple target segments determined by multiple watching users, the segment fusion processing is performed to obtain the final target segment.

[0167] The segment fusion processing can refer to the process of integrating these scattered segments into a final target segment by a specific algorithm after multiple users determine the target segments for the same target media content. The core purpose is to integrate the preferences or judgments of multiple users, so that the final result is more universal and representative, and the influence of individual differences of a single user on the selection of the segment is avoided.

[0168] In an embodiment, after the target segment satisfying the set condition is determined from the target media content according to the sequence of emotional state data, the method further includes: performing segment fusion processing based on the plurality of target segments determined by the plurality of viewing users to obtain a final target segment. The final target segment can reflect the common preferences or points of attention of the plurality of users, avoid one-sidedness caused by individual differences of a single user, and directly extract the target segment from the target media content as a highlight segment or a wonderful segment. The target segment is subsequently edited into a blooper reel, a trailer, or a promotional film, etc., thereby improving the promotional efficiency and being able to introduce more emotional resonance of users.

[0169] Specifically, an exemplary implementation of performing segment fusion processing based on the plurality of target segments determined by the plurality of viewing users to obtain a final target segment includes: obtaining one target segment determined by each viewing user, taking an average value of start frame positions of the plurality of target segments of the plurality of viewing users to obtain a fusion start frame, taking an average value of end frame positions of the plurality of target segments to obtain a fusion end frame, and determining the final target segment from the target media content according to the fusion start frame and the fusion end frame.

[0170] For example, assuming that there are three users, each of which determines one target segment for the same target media content, the target segment of user A is [10th frame-30th frame], the target segment of user B is [12th frame-32th frame], and the target segment of user C is [8th frame-28th frame]. An average value of start frame positions of the plurality of target segments is taken. The fusion start frame = (10+12+8) ÷ 3 = 10 frames. Similarly, an average value of end frame positions of the plurality of target segments is taken. The fusion end frame = (30+32+28) ÷ 3 = 30 frames. According to the calculated fusion start frame (10th frame) and the fusion end frame (30th frame), a corresponding segment [10th frame-30th frame] is extracted from the target media content, which is the final target segment.

[0171] As another possible implementation, the plurality of target segments determined by each user are obtained, target segments with overlapping parts in the plurality of users are screened and determined as overlapping target segments, segment fusion processing is performed on the overlapping target segments of the plurality of users to obtain a final target segment, and then the final wonderful segment is determined according to the plurality of target segments determined by each user and the final target segment obtained after the fusion processing.

[0172] For example, assuming that two users each determine a plurality of target segments for the same target media content, the plurality of target segments of user A includes {[6th frame-8th frame], [10th frame-30th frame]}; the target segments of user B are {[2nd frame-5th frame], [12th frame-32th frame]}. It can be seen that the overlapping target segments of user A and user B include [10th frame-30th frame], [12th frame-32th frame], and the average value of the start frame positions of the plurality of target segments is taken. The fusion start frame is (10+12)÷2=11 frames, and the average value of the end frame positions of the plurality of target segments is taken, and the fusion end frame is (30+32)÷2=31 frames. According to the calculated fusion start frame (11th frame) and fusion end frame (31th frame), the corresponding segment [11th frame-31th frame] is extracted from the target media content, which is the final target segment. The wonderful segments determined after the segment fusion processing can be {[2nd frame-5th frame], [6th frame-8th frame], [11th frame-31th frame]}, and the above wonderful segments can be directly extracted as a promotional film, etc.

[0173] In addition, other methods can also be used to fuse the plurality of target segments of a plurality of users to obtain the final target segment, for example, a large language model or big data technology can be used to analyze and process the plurality of target segments obtained according to a plurality of users, and correct errors to obtain more universal wonderful segments or highlight segments that conform to the emotional state changes of users. The embodiments of the present application do not limit this.

[0174] Through Figure 4 the related description of the embodiments shown, through the fusion processing of the plurality of target segments obtained according to the emotional data of a plurality of users, the final target segment is generated, the collective wisdom is integrated, the one-sidedness caused by individual differences is overcome, and the universality and application value of the final target segment are improved, so that the highlight segment (the final target segment) extracted from the media content is more consistent with the public cognition, and can play a greater role in content creation, promotion and other scenes.

[0175] Figure 5 A media content processing device provided by an embodiment of the present application, the device comprises:

[0176] The data acquisition module 51 is configured to acquire a face data sequence of a viewing user through a camera device in a process of playing a target media content;

[0177] The data analysis module 52 is configured to analyze a face action unit combination sequence from the face data sequence;

[0178] The emotional state determination module 53 is configured to determine an emotional state data sequence of the viewing user according to the face action unit combination sequence.

[0179] The target segment determination module 54 is configured to determine a target segment from the target media content according to the sequence of emotional state data, the target segment satisfying a set condition.

[0180] In a possible implementation, the data analysis module 52 is specifically configured to:

[0181] The following processing is performed on each piece of facial data in the sequence of facial data respectively, to obtain a sequence of facial action unit combinations:

[0182] An action intensity parameter set of a target facial feature region is analyzed from the facial data.

[0183] The action intensity parameter set is mapped to a facial action unit combination according to a preset facial action coding system rule.

[0184] In a possible implementation, the emotional state determination module 53 is specifically configured to:

[0185] The following processing is performed on each facial action unit combination in the sequence of facial action unit combinations respectively, to obtain a sequence of emotional states of the viewing user:

[0186] An emotional expectation type of a media content segment played when the facial data corresponding to the facial action unit combination is collected is determined.

[0187] According to the emotional expectation type, a weight coefficient of each facial action unit in the facial action unit combination is determined.

[0188] The facial action unit combination is quantified into emotional state data according to the weight coefficient of each facial action unit.

[0189] In a possible implementation, the target segment determination module 54 is configured to:

[0190] An emotional state data determination unit is configured to determine a plurality of target emotional state data satisfying a local extreme condition from the sequence of emotional state data.

[0191] A target frame determination unit is configured to determine a target frame corresponding to each target emotional state data from the target media content.

[0192] A target segment determination unit is configured to determine a target segment from the target media content according to a plurality of target frames, the target segment satisfying a set condition.

[0193] In a possible implementation, the emotional state data determination unit is specifically configured to:

[0194] initializing a sliding window with a length of N, and aligning a start of the sliding window with a first emotion state data of the sequence of emotion state data;

[0195] obtaining N emotion state data in a current sliding window;

[0196] marking the emotion state data at a center position of the current sliding window as a first emotion state data;

[0197] in a case where the first emotion state data is greater than other emotion state data in the current sliding window, determining the first emotion state data as a target emotion state data satisfying a local extremum condition;

[0198] sliding the current sliding window by M emotion state data along the sequence of emotion state data in a direction towards an end, and returning to perform the obtaining N emotion state data in a current sliding window and the subsequent steps until an end of the sliding window reaches an end of the sequence of emotion state data.

[0199] In a possible implementation, the target segment determining unit is specifically configured to:

[0200] sequentially traversing a plurality of the target frames, and marking a first target frame as a target segment start frame when the first target frame is traversed;

[0201] starting from a second target frame that is traversed, determining a frame interval between a currently traversed target frame and a previous target frame of the currently traversed target frame, and marking the currently traversed target frame as a target segment start frame and marking the previous target frame as a target segment end frame in a case where the frame interval is greater than a set frame interval threshold;

[0202] determining a target segment satisfying a set condition from the target media content based on adjacent target segment start frames and target segment end frames.

[0203] In a possible implementation, the apparatus further includes:

[0204] a fusion processing module configured to perform segment fusion processing based on a plurality of target segments determined by a plurality of the watching users to obtain a final target segment.

[0205] In a possible implementation, the fusion processing module is specifically configured to:

[0206] averaging start frame positions of the plurality of target segments to obtain a fusion start frame, and averaging end frame positions of the plurality of target segments to obtain a fusion end frame;

[0207] According to the fusion start frame and the fusion end frame, a final target segment is determined from the target media content.

[0208] As shown in Figure 6 The embodiments of the present application provide an electronic device, which comprises a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112 and the memory 113 complete mutual communication through the communication bus 114,

[0209] The memory 113 is used for storing a computer program.

[0210] In an embodiment of the present application, the processor 111 is used for executing the program stored in the memory 113, and a media content processing method provided by any one of the foregoing method embodiments is implemented.

[0211] In the process of playing the target media content, the face data sequence of the watching user is collected through the camera device;

[0212] The face action unit combination sequence is parsed from the face data sequence;

[0213] According to the face action unit combination sequence, the emotional state data sequence of the watching user is determined;

[0214] According to the emotional state data sequence, the target segment meeting the set condition is determined from the target media content.

[0215] The embodiments of the present application further provide a computer readable storage medium, which has a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the media content processing method provided by any one of the foregoing method embodiments.

[0216] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. According to actual needs, some or all of the modules can be selected to achieve the purpose of the embodiments of the present application.

[0217] Those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can be implemented by hardware. Based on such an understanding, the technical solutions described above essentially or in other words make contributions to the related art, and can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, or an optical disk, and includes a plurality of instructions used to cause a computer device (such as a personal computer, a server, or a network device) to execute the methods described in each embodiment or some parts of the embodiments.

[0218] It is to be understood that the terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes", "including" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order in which they are described unless specifically indicated as such. It is also to be understood that additional or alternative steps can be employed.

[0219] The above description is merely that of specific embodiments of the present application, and thus is not intended to limit the present application. Various modifications to these embodiments can be apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Accordingly, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A media content processing method, characterized in that: The method comprises: During the playback of the target media content, a facial data sequence of the viewing user is collected by a camera device; Parsing a facial action unit combination sequence from the facial data sequence; determining an emotional state data sequence of the viewing user according to the facial action unit combination sequence; According to the emotional state data sequence, a target segment that meets a set condition is determined from the target media content.

2. The method according to claim 1, characterized in that The step of parsing the facial action unit combination sequence from the facial data sequence includes: The following processing is performed on each facial data in the facial data sequence to obtain a facial action unit combination sequence: Analyzing the action intensity parameter set of the target facial feature area from the facial data; According to a preset facial action coding system rule, the action intensity parameter set is mapped into a facial action unit combination.

3. The method according to claim 1, characterized in that Determining the emotional state data sequence of the viewing user based on the facial action unit combination sequence includes: The following processing is performed on each facial action unit combination in the facial action unit combination sequence to obtain the emotional state sequence of the viewing user: determining an expected emotion type of a media content segment played when facial data corresponding to the facial action unit combination is collected; Determining a weight coefficient of each facial action unit in the facial action unit combination according to the expected emotion type; The facial action units are combined and quantified into emotional state data according to the weight coefficients of the facial action units.

4. The method according to claim 1, wherein The step of determining a target segment that meets a set condition from the target media content according to the emotional state data sequence includes: Determining a plurality of target emotional state data satisfying local extreme value conditions from the emotional state data sequence; Determining, from the target media content, a target frame corresponding to each target emotional state data; A target segment meeting a set condition is determined from the target media content according to the plurality of target frames.

5. The method according to claim 4, characterized in that Determining a plurality of target emotional state data satisfying local extreme value conditions from the emotional state data sequence includes: Initializing a sliding window of length N, and aligning the starting end of the sliding window with the first emotional state data of the emotional state data sequence; Get N emotional state data within the current sliding window; Marking the emotional state data located at the center of the current sliding window as first emotional state data; In a case where the first emotional state data is greater than other emotional state data in the current sliding window, determining the first emotional state data as target emotional state data that meets a local extreme value condition; Slide the current sliding window along the emotional state data sequence toward the end by M emotional state data, and return to execute the steps of obtaining N emotional state data in the current sliding window and subsequent steps until the end of the sliding window reaches the end of the emotional state data sequence.

6. The method according to claim 4, characterized in that The determining a target segment that meets a set condition from the target media content according to the plurality of target frames includes: traversing the plurality of target frames in sequence, and when a first target frame is reached, marking the first target frame as a target segment start frame; Starting from the second target frame traversed, determining the frame interval between the currently traversed target frame and the previous target frame; if it is determined that the frame interval is greater than the set frame interval threshold, marking the currently traversed target frame as the target segment start frame, and marking the previous target frame as the target segment end frame; Based on the adjacent target segment start frame and the target segment end frame, a target segment that meets a set condition is determined from the target media content.

7. The method according to claim 1, characterized in that The method further comprises: Based on the multiple target segments determined by the multiple viewing users, segment fusion processing is performed to obtain a final target segment.

8. The method according to claim 7, characterized in that The performing segment fusion processing based on the multiple target segments determined by the multiple viewing users to obtain the final target segment includes: Taking an average of the start frame positions of the multiple target segments to obtain a fusion start frame; taking an average of the end frame positions of the multiple target segments to obtain a fusion end frame; A final target segment is determined from the target media content according to the fusion start frame and the fusion end frame.

9. A media content processing device, characterized in that: The device comprises: A data acquisition module is used to collect facial data sequences of viewing users through a camera device during the playback of target media content; A data parsing module, configured to parse a facial action unit combination sequence from the facial data sequence; an emotional state determination module, configured to determine an emotional state data sequence of the viewing user based on the facial action unit combination sequence; The target segment determination module is used to determine a target segment that meets a set condition from the target media content according to the emotional state data sequence.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is configured to execute a media content processing program stored in the memory to implement the media content processing method according to any one of claims 1 to 7.

11. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the media content processing method according to any one of claims 1 to 7.

Images