Media file processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a media file processing method and device, electronic equipment and a storage medium. The method comprises the steps that a first image frame in a media file is displayed; wherein the first image frame comprises a first object and a second object, and the first object in the first image frame is highlighted compared with the second object; in response to the detected object switching instruction, highlighting a second object in the first image frame; wherein the sound of the second object in the first image frame is highlighted so as to be matched with the highlighted second object. Through the method, the highlighted main body can be dynamically matched with the main body to which the highlighted sound belongs, and the audio-visual effect of the media file is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of media file processing, and in particular, to a media file processing method, apparatus, electronic device, and storage medium. Background Art

[0002] In the processes of recording and post-processing media files such as audio-visuals and dynamic photo albums, users may adjust the focus subject in the video frame by adjusting the focal length, shooting angle, etc., so as to prominently present the focus subject. However, after adjusting the image frame in the media file to highlight the focus subject, there is a situation where the prominent sound does not match the focus subject, resulting in a poor presentation effect of the media file. Summary of the Invention

[0003] The present disclosure provides a media file processing method, apparatus, electronic device, and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a media file processing method, including:

[0005] Displaying a first image frame in the media file; wherein, the first image frame includes a first object and a second object, and the first object in the first image frame is prominently displayed compared with the second object;

[0006] In response to detecting an object switching instruction, prominently displaying the second object in the first image frame; wherein, the sound of the second object in the first image frame is prominent to match the prominently displayed second object.

[0007] In some embodiments, the prominently displaying the second object in the first image frame includes:

[0008] Displaying a first mark in the first image frame to prominently display the second object; wherein, the first mark is located in the area where the second object is located.

[0009] In some embodiments, the method further includes:

[0010] Displaying a second mark in the first image frame; wherein, the second mark is located in the area where the first object is located; the visual effect of the second mark is different from that of the first mark.

[0011] In some embodiments, the prominently displaying the second object in the first image frame includes:

[0012] By adjusting the display effect of the area where the second object is located, and / or,

[0013] Adjusting the display effect of the area other than the second object to prominently display the second object.

[0014] In some embodiments, highlighting the second object by adjusting the display effect of the area where the second object is located, and / or adjusting the display effect of the area outside the second object includes:

[0015] Highlighting the second object by adjusting the pixel values of the area where the second object is located, and / or the pixel values of the area outside the second object.

[0016] In some embodiments, highlighting the second object by adjusting the pixel values of the pixel area outside the second object includes:

[0017] In response to detecting a blurring instruction, blurring the pixel area outside the second object in the first image frame.

[0018] In some embodiments, the method further includes:

[0019] Enhancing the sound of the second object, and / or

[0020] Eliminating or reducing the sound of the first object to highlight the sound of the second object.

[0021] In some embodiments, the method further includes:

[0022] In response to detecting the object switching instruction, identifying the first image frame affected by the object switching instruction as a key frame;

[0023] Highlighting the second object in the first image frame includes:

[0024] Highlighting the second object in each image frame between the key frame and the next key frame; wherein, the sound of the second object in each image frame is prominent.

[0025] In some embodiments, the method further includes:

[0026] In response to detecting an operation instruction for a first axis control with a key frame as a node, displaying a control indicating reset;

[0027] In response to detecting an operation instruction for the control indicating reset, resetting the identification of the image frame indicated by the operation instruction for the first axis control as a non-key frame.

[0028] In some embodiments, the method further includes:

[0029] In response to detecting an operation instruction for a second axis control with the first image frame as a node, previewing the target image frame indicated by the operation instruction for the second axis control;

[0030] Highlighting the second object in the first image frame includes:

[0031] Highlighting the second object in the currently previewed target image frame; wherein, the sound of the second object in the target image frame is prominent.

[0032] In some embodiments, the method further includes:

[0033] In response to detecting a play instruction, playing the second image frame before the switching object and the first image frame highlighting the second object after the switching object; wherein, the sound of the second object in the first image frame is prominent.

[0034] In some embodiments, the method further includes:

[0035] In response to detecting a cancel instruction, canceling the highlighting of the second object; wherein, the prominent sound of the second object is canceled.

[0036] In some embodiments, the method further includes:

[0037] In response to detecting a first save instruction, saving the second image frame before the switching object and the first image frame highlighting the second object after the switching object; wherein, the sound of the second object in the first image frame is prominent.

[0038] In some embodiments, the method further includes:

[0039] In response to detecting a second save instruction, saving the second image frame before the switching object and the image of the second object in the first image frame after the switching object.

[0040] According to a second aspect of the embodiments of the present disclosure, there is provided a media file processing device, including:

[0041] A first display module for displaying a first image frame in the media file; wherein, the first image frame includes a first object and a second object, and the first object in the first image frame is highlighted compared to the second object;

[0042] A switching module for, in response to detecting an object switching instruction, highlighting the second object in the first image frame; wherein, the sound of the second object in the first image frame is prominent to match the highlighted second object.

[0043] In some embodiments, the switching module is configured to display a first mark in the first image frame to highlight the second object; wherein, the first mark is located in the area where the second object is located.

[0044] In some embodiments, the apparatus further comprises:

[0045] A second display module, configured to display a second marker in the first image frame; wherein, the second marker is located in the area where the first object is located; and the visual effect of the second marker is different from that of the first marker.

[0046] In some embodiments, the switching module is configured to highlight the second object by adjusting the display effect of the area where the second object is located, and / or adjusting the display effect of the area outside the second object.

[0047] In some embodiments, the switching module is configured to highlight the second object by adjusting the pixel values of the area where the second object is located, and / or the pixel values of the area outside the second object.

[0048] In some embodiments, the switching module is configured to blur the pixel area outside the second object in the first image frame in response to detecting a blurring instruction.

[0049] In some embodiments, the apparatus further comprises:

[0050] An adjustment module, configured to enhance the sound of the second object, and / or eliminate or reduce the sound of the first object, so as to highlight the sound of the second object.

[0051] In some embodiments, the apparatus further comprises:

[0052] A marking module, configured to identify the first image frame acted on by the object switching instruction as a key frame in response to detecting the object switching instruction;

[0053] The switching module is configured to highlight the second object in each image frame between the key frame and the next key frame; wherein, the sound of the second object in each image frame is prominent.

[0054] In some embodiments, the apparatus further comprises:

[0055] A third display module, configured to display a control for indicating reset in response to detecting an operation instruction for a first axis control with a key frame as a node;

[0056] A reset module, configured to reset the identification of the image frame indicated by the operation instruction for the first axis control as a non-key frame in response to detecting an operation instruction for the control for indicating reset.

[0057] In some embodiments, the apparatus further comprises:

[0058] A preview module, configured to, in response to detecting an operation instruction for a second-axis control with the first image frame as a node, preview a target image frame indicated by the operation instruction for the second-axis control;

[0059] The switching module is configured to highlight a second object in the currently previewed target image frame; wherein, the sound of the second object in the target image frame is prominent.

[0060] In some embodiments, the apparatus further includes:

[0061] A playing module, configured to, in response to detecting a playing instruction, play a second image frame before the switching object and a first image frame after the switching object that highlights the second object; wherein, the sound of the second object in the first image frame is prominent.

[0062] In some embodiments, the apparatus further includes:

[0063] A canceling module, configured to, in response to detecting a canceling instruction, cancel the highlighting of the second object; wherein, the prominent sound of the second object is canceled.

[0064] In some embodiments, the apparatus further includes:

[0065] A first saving module, configured to, in response to detecting a first saving instruction, save the second image frame before the switching object and the first image frame after the switching object that highlights the second object; wherein, the sound of the second object in the first image frame is prominent.

[0066] In some embodiments, the apparatus further includes:

[0067] A second saving module, configured to, in response to detecting a second saving instruction, save the second image frame before the switching object and the image of the second object in the first image frame after the switching object.

[0068] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0069] A processor; a memory for storing processor-executable instructions; wherein, the processor is configured to execute the method as described in the first aspect above.

[0070] According to a fourth aspect of the embodiments of the present disclosure, there is provided a storage medium, including:

[0071] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in the first aspect above.

[0072] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0073] In the case where the processing device according to the embodiment of the present disclosure includes a first image frame with a highlighted first object, in response to detecting an object switching instruction, a second object in the first image frame is highlighted, and the sound of the second object is highlighted, so as to dynamically match the subject highlighted in the first image frame with the subject to which the highlighted sound belongs, thereby improving the audiovisual effect of the media file.

[0074] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Brief Description of the Drawings

[0075] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0076] Figure 1 It is a schematic diagram of an example of a video processing method in the related art.

[0077] Figure 2 It is a schematic diagram of a process example of a media file processing method shown in the embodiment of the present disclosure.

[0078] Figure 3 It is a schematic diagram of an object focusing function shown in the embodiment of the present disclosure.

[0079] Figure 4 It is a schematic diagram of an example of switching the focus subject shown in the embodiment of the present disclosure.

[0080] Figure 5 It is a schematic diagram of an example of resetting key frames shown in the embodiment of the present disclosure.

[0081] Figure 6 It is a schematic diagram of an example of switching the object focusing function shown in the embodiment of the present disclosure.

[0082] Figure 7 It is a schematic diagram of a media file processing device shown in the embodiment of the present disclosure.

[0083] Figure 8 It is a block diagram of an electronic device shown in the embodiment of the present disclosure. Detailed Embodiments

[0084] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0085] Figure 1 Shown is an example diagram of a video processing method in the related art. Figure 1 In this method, the sound source position is judged and the video scene is analyzed through the scene analysis technology of multimodal audio; high-quality audio is extracted based on deep learning technology, including technologies such as multimodal speech enhancement and model beamforming. In addition, after the audio is extracted, the gain synchronization of the sound and the picture and the volume balance of multiple sound sources can be performed to ensure that the quality of the audio does not decrease compared to the original video.

[0086] However, the above solution does not involve the recognition of the focus subject in the audio-visual and the matching of the focus subject in the highlighted video frame and the sound of the focus subject, resulting in poor presentation effects of the audio-visual.

[0087] In response to this, an embodiment of the present disclosure provides a media file processing method, and its execution subject may be a media file processing device (abbreviated as a processing device). For example, the media file processing method may be executed by a terminal device or other electronic devices. Among them, the terminal device may be a device with video recording functions such as a mobile phone, a camera, a tablet computer, a vehicle-mounted device, and a wearable device. In some possible implementation manners, the media file processing method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0088] Figure 2 is an example flowchart of a media file processing method shown in an embodiment of the present disclosure. As Figure 2 can be seen, it includes the following steps:

[0089] S21. Display a first image frame in the media file; wherein, the first image frame includes a first object and a second object, and the first object in the first image frame is highlighted compared to the second object;

[0090] S22. In response to detecting an object switching instruction, highlight the second object in the first image frame; wherein, the sound of the second object in the first image frame is prominent to match the highlighted second object.

[0091] In an embodiment of the present disclosure, media files such as audio-visual, dynamic photos, etc. include multiple image frames, as well as audio synchronized in time for each image frame; the image frames and the corresponding audio in the embodiment of the present disclosure may be obtained by real-time acquisition during the audio-visual recording process, or may be media files with associated audio generated based on existing image frames in a dynamic photo album.

[0092] Taking media files as an example of audio and video, the audio and video in the embodiments of the present disclosure may be recorded in a preset mode, such as recorded in a standard mode, a movie mode, a portrait mode, etc.; the scenarios of video recording such as interviews, family gatherings, ball games, skiing, etc., are not limited in the embodiments of the present disclosure.

[0093] In step S21, the processing device displays the first image frame in the media file through a display component (such as a display screen); wherein, the first object may be one or more objects, and the second object may also be one or more objects; the first object and the second object are not limited to people, animals, electronic devices that support sound output, etc.

[0094] In the embodiments of the present disclosure, the first object in the displayed first image frame is highlighted compared with the second object. For example, by displaying a mark for identifying the area where the first object is located to highlight the first object; or, by adjusting the display resolution of the first object and / or the second object so that the display resolution of the first object is greater than that of the second object to highlight the first object; it may also be by superimposing a mask on the second object to highlight the first object, etc. The embodiments of the present disclosure do not limit the specific form of highlighting the first object compared with the second object in the first image frame.

[0095] In step S22, when the processing device detects an object switching instruction, it highlights the second object in the first image frame; wherein, the processing device detects the object switching instruction, which may be a touch instruction, a voice instruction, a gesture instruction, etc. issued by the detected user. For example, the user clicks on the object in the previewed first image frame, and determines that the object indicated by the click operation is the second object; the user may determine that the object affected by the sliding gesture representing magnification triggered on the object in the previewed first image frame is the second object.

[0096] In addition, the processing device may also identify the object switching instruction in an automatic manner; for example, in the embodiments of the present disclosure, the processing device determines the area ratio of each object in the first image frame, and in response to the change of the object with the largest area ratio, detects the object switching instruction, etc., and uses the object with the largest area ratio as the second object indicated by the object switching instruction. Or, based on the change of the object detected within the preset center range of the first image frame, the object detected within the current preset center range is used as the second object indicated by the object switching instruction, etc.

[0097] Embodiments of the present disclosure can highlight a second object in a manner similar to highlighting the first object described above. For example, the second object can be highlighted by displaying a marker for identifying the area where the second object is located; or, by adjusting the display resolution of the first object and / or the second object so that the display resolution of the second object is greater than that of the first object to highlight the second object; or, by superimposing a mask on the area outside the second object to highlight the second object. Embodiments of the present disclosure do not limit the specific form of highlighting the second object in the first image frame.

[0098] When embodiments of the present disclosure highlight the second object in response to detecting an object switching instruction, the sound of the first object and / or the second object in the first image frame is adjusted so that the sound of the second object is more prominent, thereby matching the highlighted second object.

[0099] In embodiments of the present disclosure, since the first image frame corresponds to the sound of the first object and the sound of the second object; in some embodiments, the processing device can, based on the preset correspondence between the first object and the sound of the first object, and the correspondence between the second object and the sound of the second object, highlight the sound of the second object when highlighting the second object.

[0100] In some embodiments, the media file is further associated with position information, including the spatial positions of the sound-emitting subjects in the audio corresponding to the image frame; among them, the spatial positions of the sound-emitting subjects in the audio can be represented by coordinates in the world coordinate system, coordinates in the coordinate system of the media file generation device, or can also be represented by the sound reception angle in the audio space formed by the audio of each sound-emitting subject.

[0101] In embodiments of the present disclosure, the audio of each sound-emitting subject in the audio frame can be recognized in the initial audio by processing the initial audio through algorithms such as the Mel frequency cepstrum algorithm, a trained recurrent neural network (Recurrent Neural Networks), and / or a convolutional neural network, so as to separate and identify the audio of different sound-emitting subjects recognized from the initial audio. In addition, the frequency band corresponding to each sound-emitting subject can also be determined and identified in the initial audio according to the mapping relationship between the pre-saved voiceprint (such as the sound frequency band) and the sound-emitting subject, as the audio of each sound-emitting subject. Thus, after recognizing the audio of each sound-emitting subject in the audio frame, based on the azimuth of each audio, the sound reception angle corresponding to each sound-emitting subject is determined. In addition, after identifying each target object in the video frame based on algorithms such as the target detection algorithm and depth calculation, based on the correspondence between the position in the video frame and the sound reception angle, the coordinates of the sound-emitting subject in the coordinate system of the audio-video recording device can be obtained; based on the conversion relationship between the coordinate system of the audio-video recording device and the position in the video frame, the coordinates of each sound-emitting subject in the world coordinate system can also be obtained.

[0102] In some embodiments, taking the spatial position of the sound - emitting body characterized by the sound - receiving angle as an example, the processing device converts the position of the second object in the first image frame based on a preset conversion equation to obtain a target sound - receiving angle, and determines the audio collected at the sound - receiving angles falling within a preset angle range centered on the target sound - receiving angle as the sound of the second object; wherein the preset conversion equation is obtained by fitting based on multiple tests.

[0103] After determining the sound of the second object in the embodiments of the present disclosure, noise reduction processing, voice separation, etc. can be performed on the sound of the second object to make the sound of the second object clearer; due to the interference of noise (such as ambient sound) on the sound of the first object, the sound of the second object after noise reduction processing is more easily captured by the user, thus highlighting the sound of the second object.

[0104] It should be noted that the audio - video (including image frames and the audio corresponding to the image frames) and the position information of the associated audio - video in the embodiments of the present disclosure can be encapsulated into a media file, so that after obtaining the media file subsequently, editing processing can be directly performed based on the audio - video and the position information of the associated audio - video in the media file. While highlighting the second object, the sound of the second object is highlighted to achieve "object focusing".

[0105] After highlighting the sound of the second object in the embodiments of the present disclosure, the first image frame including the highlighted second object is associated with the audio after highlighting the sound of the second object based on a time - stamp or other means to obtain a processed media file.

[0106] In addition, image frame clipping, adding a preset video template, mixing with other audio - video, superimposing filters, aspect ratio adjustment, etc. can be performed on the processed audio - video to obtain a final media file.

[0107] In some embodiments, highlighting the second object in the first image frame includes:

[0108] Displaying a first mark in the first image frame to highlight the second object; wherein the first mark is located in the area where the second object is located.

[0109] In the embodiments of the present disclosure, in response to detecting an object switching instruction, a first marker for identifying the area where the second object is located is displayed in the first image frame. For example, the first marker is located in the area where the head of the second object is located, the area of the upper body, the area of the entire torso, etc. Among them, the first marker can highlight the second object in the first image frame, and the presence of the first marker does not affect the content of the first image frame. If the first image frame in the embodiments of the present disclosure only includes one layer, the first marker can be displayed by overlaying a mask on the first image frame. If the first image frame includes multiple layers, the first marker can be displayed on the layer above the layer where the image of the first image frame is located, so as to bring an overall display effect to the user.

[0110] The shape of the first marker in the embodiments of the present disclosure can be a square, a rectangle, a circle, other polygons, etc. The line of the first marker can be a solid line, a dotted line, etc. The color of the first marker can be any color. The embodiments of the present disclosure do not limit the specific style of the first marker.

[0111] It can be understood that in the embodiments of the present disclosure, by displaying the first marker for identifying the area where the second object is located, the user can intuitively learn the current prominent focus object through the displayed first marker, and the effect of highlighting the second object is better.

[0112] In some embodiments, the method further includes:

[0113] Displaying a second marker in the first image frame; wherein, the second marker is located in the area where the first object is located; the visual effect of the second marker is different from the visual effect of the first marker.

[0114] In the embodiments of the present disclosure, while displaying the first marker in the first image frame, a second marker for identifying the area where the first object is located can also be displayed; among them, the visual effect of the second marker is different from the visual effect of the first marker. For example, their shapes, sizes, colors, etc. are different, and the first marker is more prominent visually than the second marker. Exemplarily, the size of the first marker is larger than the size of the second marker; the line color of the first marker is yellow, and the line color of the second marker is gray, etc., so that the image in the area where the first marker is located (that is, highlighting the second object) can be highlighted based on the displayed first marker and second marker.

[0115] It can be understood that in the case where the first image frame in the embodiments of the present disclosure includes multiple objects, based on the different markers corresponding to the first object and the second object, the first object is highlighted, which is convenient for the user to intuitively learn the current focus object. In addition, in the case where the user intends to switch the focus object, the focus object can be switched conveniently and quickly by, for example, clicking on the area where the first marker is located.

[0116] In some embodiments, highlighting the second object in the first image frame includes:

[0117] By adjusting the display effect of the area where the second object is located, and / or,

[0118] Adjusting the display effect of the area outside the second object to highlight the second object.

[0119] In the embodiments of the present disclosure, in response to detecting an object switching instruction, the display effect of the area where the second object is located, and / or the display effect of the area outside the second object is adjusted by overlaying a mask layer to highlight the second object; for example, overlaying a mask layer in the area where the first object is located to highlight the second object; or, by overlaying mask layers with different transparencies in the area where the first object is located and the area where the second object is located to highlight the second object, wherein the transparency of the mask layer overlaid in the area where the first object is located is less than the transparency of the mask layer overlaid in the area where the second object is located. In addition, it can also be overlaying a mask layer with a preset image (such as a star, a flame, etc.) drawn in the area where the second object is located, and highlighting the second object through the preset image that is more attractive to the user's attention. Through this solution, the second object can be highlighted in a more flexible and variable form to meet the diverse needs of users.

[0120] In some embodiments, the highlighting the second object by adjusting the display effect of the area where the second object is located, and / or adjusting the display effect of the area outside the second object includes:

[0121] By adjusting the pixel values of the area where the second object is located, and / or the pixel values of the area outside the second object, to highlight the second object.

[0122] In the embodiments of the present disclosure, the processing device can adjust the pixel values of the area where the second object is located to highlight the second object; for example, by increasing the pixel brightness of each pixel point in the area where the second object is located to highlight the second object; or, increasing the contrast of the pixel points in the area where the second object is located to highlight the second object; it can also change the edge color of the second object (such as yellow) to highlight the second object, etc.

[0123] And / or, the processing device highlights the second object by adjusting the pixel values of the area outside the second object; for example, by reducing the pixel brightness of each pixel point in the area outside the second object to highlight the second object; or, reducing the contrast of the pixel points in the area outside the second object to perform a mosaic process on the area outside the second object visually, thereby highlighting the second object; it can also change the color of the area outside the second object (such as turning it gray) to highlight the second object, etc.

[0124] It can be understood that in the embodiments of the present disclosure, there is no need to overlay layers. By adjusting the pixel values of the area where the second object is located and / or the pixel values of the area outside the second object, the second object is highlighted, which has the advantages of being simple and easy to implement.

[0125] In some embodiments, the highlighting of the second object by adjusting the pixel values of the pixel area outside the second object includes:

[0126] In response to detecting a blurring instruction, blurring the area outside the second object in the first image frame.

[0127] In the embodiments of the present disclosure, the processing device detecting the blurring instruction may be detecting a touch instruction, a voice instruction, a gesture instruction, etc. issued by the user. In addition, the processing device may also detect the blurring instruction in an automatic manner; for example, when the object indicated by the object switching instruction is inconsistent with the currently highlighted object, the blurring instruction is detected, so as to blur the area outside the second object in the first image frame to highlight the second object. Among them, the blurring processing of the area outside the second object can be realized by a Gaussian blur algorithm, a bokeh blur algorithm, an aperture blur algorithm, etc.

[0128] It can be understood that the embodiments of the present disclosure can blur the area outside the second object based on detecting the blurring instruction issued by the user to meet the user's need to highlight the second object; in addition, the area outside the second object can also be blurred by improving the way of automatically triggering the blurring instruction, improving the intelligence of highlighting the second object.

[0129] In some embodiments, the method further includes:

[0130] Enhancing the sound of the second object, and / or,

[0131] Eliminating or reducing the sound of the first object to highlight the sound of the second object.

[0132] In the embodiments of the present disclosure, in response to detecting an object switching instruction, the processing device highlights the sound of the second object by enhancing the sound of the second object, such as increasing a preset volume value based on the sound volume of the second object, increasing the sound volume of the second object to a preset multiple, etc.; or enhancing the sound of the second object to the volume intended by the user based on the detected user operation, etc.

[0133] And / or,

[0134] In response to detecting an object switching instruction, the processing device highlights the sound of a second object by eliminating or reducing the sound of a first object; wherein, reducing the sound of the first object, such as reducing a preset volume value based on the volume of the sound of the first object, reducing the volume of the sound of the second object to a preset multiple, etc.; or, reducing the sound of the first object to the volume intended by the user based on the detected user operation, thereby highlighting the sound of the second object. Through this solution, the sound of the second object can be highlighted in a more flexible and variable manner to meet the diverse needs of users.

[0135] It can be understood that in the embodiment of the present disclosure, when the processing device includes a first image frame with a highlighted first object, in response to detecting an object switching instruction, the second object in the first image frame is highlighted, and the sound of the second object is highlighted, so as to dynamically match the main body highlighted in the first image frame with the main body to which the highlighted sound belongs, thereby improving the audio-visual effect of the media file.

[0136] In some embodiments, the method further includes:

[0137] In response to detecting the object switching instruction, identifying the first image frame on which the object switching instruction acts as a key frame;

[0138] The highlighting the second object in the first image frame includes:

[0139] Highlighting the second object in each image frame between the key frame and the next key frame; wherein, the sound of the second object in each of the image frames is highlighted.

[0140] In the embodiment of the present disclosure, when the processing device detects an object switching instruction, it identifies the first image frame on which the object switching instruction acts as a key frame; after identifying the key frame, it highlights the second object in each image frame between the key frame and the next key frame, and highlights the sound of the second object in each image frame between the key frame and the next key frame.

[0141] It can be understood that the embodiment of the present disclosure focuses on the second object in each image frame between the key frame and the next key frame as a whole (highlighting the second object and highlighting the sound of the second object). Compared with enhancing the sound of the second object in each image frame one by one, the embodiment of the present disclosure can improve the coherence of the display and sound focusing of the second object in multiple image frames within a continuous time range.

[0142] In some embodiments, the method further includes:

[0143] In response to detecting an operation instruction on a first axis control with a key frame as a node, displaying a control indicating reset;

[0144] In response to detecting an operation instruction for the control indicating reset, the image frame reset indicated by the operation instruction for the first-axis control is marked as a non-key frame.

[0145] In the embodiments of the present disclosure, the preview interface of the media file includes a first-axis control with key frames as nodes; among them, each node in the first-axis control corresponds to a key frame, and the arrangement of the nodes on the axis can be randomly arranged, equally spaced arranged; or arranged at intervals based on the chronological order corresponding to the key frames, so that the first-axis control can be regarded as a time-axis control of the key frames.

[0146] When the processing device in the embodiments of the present disclosure detects an operation instruction for a node on the first-axis control, it displays a control indicating reset. Thus, after the user issues an operation instruction (such as clicking) for the control indicating reset, the image frame reset indicated by the operation instruction for the first-axis control is marked as a non-key frame, that is, the image frame indicated by the operation instruction for the first-axis control is not used as a key frame, so that the object (focusing subject) highlighted in this image frame is reset from the second object to the first object.

[0147] It can be understood that the processing device in the embodiments of the present disclosure, based on the first-axis control displayed on the preview interface, realizes the reset of the focusing subject in any key frame based on the first-axis control, with high flexibility and can better meet the user's personalized editing needs.

[0148] In some embodiments, the method further includes:

[0149] In response to detecting an operation instruction for a second-axis control with the first image frame as a node, preview the target image frame indicated by the operation instruction for the second-axis control;

[0150] The highlighting of the second object in the first image frame includes:

[0151] Highlight the second object in the currently previewed target image frame; among them, the sound of the second object in the target image frame is prominent.

[0152] In the embodiments of the present disclosure, the preview interface of the media file includes a second-axis control with image frames as nodes; among them, each node in the second-axis control corresponds to an image frame, and the arrangement of the nodes on the axis can be randomly arranged, equally spaced arranged; or arranged at intervals based on the chronological order corresponding to the image frames, so that the second-axis control can be regarded as a time-axis control of the image frames.

[0153] After the processing device according to an embodiment of the present disclosure detects an operation instruction for a node on the second-axis control, it previews an image frame indicated by the operation instruction for the node on the second-axis control, so that the previewed image frame serves as the current image frame to be processed, highlights a second object in the image frame, and highlights the sound of the second object.

[0154] It can be understood that the processing device according to an embodiment of the present disclosure, based on the second-axis control displayed on the preview interface, realizes highlighting a second object in any image frame based on the second-axis control and matching the sound highlighting the second object, improving the purposefulness and flexibility of the user's intention to focus on the second object.

[0155] In some embodiments, the method further includes:

[0156] In response to detecting a play instruction, play the second image frame before switching the object and the first image frame highlighting the second object after switching the object; wherein, the sound of the second object in the first image frame is prominent.

[0157] In an embodiment of the present disclosure, the preview interface of the media file includes a control indicating play, such as a text control displaying "Play", a button control displaying an icon representing play, etc.

[0158] The processing device according to an embodiment of the present disclosure, based on the preview interface, detects a play instruction for the control indicating play, plays the second image frame before switching the object and the first image frame highlighting the second object after switching the object; wherein, the sound of the second object in the first image frame is prominent, so that the user can determine whether to continue editing to meet the user's needs after browsing the played media file.

[0159] In some embodiments, the method further includes:

[0160] In response to detecting a cancel instruction, cancel highlighting the second object; wherein, the sound highlighting of the second object is cancelled.

[0161] In an embodiment of the present disclosure, the preview interface of the media file includes a control indicating cancel, such as a text control displaying "Cancel", a button control displaying an icon representing canceling the edit, etc.

[0162] The processing device according to an embodiment of the present disclosure, based on the preview interface, detects a cancel instruction for the control indicating cancel, cancels highlighting the second object in the first image frame, and restores the sound of the second object, thereby restoring the original audio and video and reducing the mis-editing of the media file.

[0163] In some embodiments, the method further includes:

[0164] In response to detecting a first save instruction, save the second image frame before the switching object and the first image frame that highlights the second object after the switching object; wherein, the sound of the second object in the first image frame is prominent.

[0165] In an embodiment of the present disclosure, the preview interface of the media file includes a first save control indicating saving, such as a text control displaying "Save", a button control displaying an icon representing saving, and the like.

[0166] The processing device in the embodiment of the present disclosure detects a first save instruction for the first save control based on the preview interface, and saves the second image frame before the switching object and the first image frame that highlights the second object after the switching object; wherein, the sound of the second object in the first image frame is prominent, so that the user can store the edited media file in a preset space (such as an album application) for subsequent multiple playbacks, sharing, uploading to the cloud, etc.

[0167] In some embodiments, the method further includes:

[0168] In response to detecting a second save instruction, save the second image frame before the switching object and the image of the second object in the first image frame after the switching object.

[0169] In an embodiment of the present disclosure, the preview interface of the media file includes a second save control indicating split saving, such as a text control displaying "Split Save", a button control displaying an icon representing split saving, and the like.

[0170] The processing device in the embodiment of the present disclosure detects a second save instruction for the second save control based on the preview interface, and saves the second image frame before the switching object and the image of the second object in the first image frame after the switching object; wherein, the image of the second object in the first image frame after the switching object may be an image obtained by cropping the second object in the first image frame, and the cropped image is only associated with the sound of the second object. Through this solution, the user can save the image and sound of only the object of interest (such as the second object), reducing the interference of irrelevant objects appearing in the picture and / or the sound of irrelevant objects.

[0171] Figure 3 It is an example diagram of an object focusing function shown in an embodiment of the present disclosure. Figure 3Among them, P1 indicates the preview interface of a media file (such as audio and video), P2 indicates the editing interface of the media file, L3 indicates the control for indicating to turn on or off the object focusing function in the preview interface of the media file, L4 indicates the control for indicating to turn on or off the object focusing function in the editing interface of the media file, L5 indicates the axis control with image frames as nodes, corresponding to the aforementioned second axis control; L6 indicates the axis control with key frames as nodes, corresponding to the aforementioned first axis control; L7 indicates the control for indicating blurring; L8 indicates the control for indicating play; L9 indicates the control for indicating cancellation; L10 indicates the control for indicating saving, corresponding to the aforementioned first saving control.

[0172] In the embodiment of the present disclosure, the user can turn on the object focusing function by clicking the control indicated by L3 in the interface indicated by P1, or by clicking the control indicated by L4 in the interface indicated by P2. After the user turns on the object focusing function, if the highlighted object is the first object at this time, the first object is highlighted by displaying a third marker for indicating the area where the first object is located; among them, a fourth marker for indicating the area where the second object is located can also be displayed, and the visual effect of the fourth marker is different from that of the third marker. For example, the line color of the third marker is yellow, and the line color of the fourth marker is gray. As shown in the figure, a1 corresponds to the third marker, and a2 corresponds to the fourth marker, so as to highlight the first object in the area indicated by a1. In addition, the control indicated by L11 indicates undo editing, that is, after clicking this control, the current editing step is undone; the control indicated by L12 indicates repeat editing, that is, after clicking this control, the current editing step is repeated once. L13 indicates time information, where the previous time represents the time of the currently previewed image frame, and the latter time represents the total duration of the media file.

[0173] Figure 4 This is an example diagram of switching the focus subject shown in the embodiment of the present disclosure. In the interface indicated by P2, as described above, the focus subject in the interface is the first object in the area indicated by a1; if the user clicks the second object indicated by L17, an object switching instruction is detected, and a first marker indicated by a4 is displayed to highlight the second object; in addition, a second marker indicated by a3 can also be displayed, and the visual effect of the first marker is different from that of the second marker. For example, the line color of the first marker is yellow, and the line color of the second marker is gray, so as to highlight the second object in the area indicated by a4. In the embodiment of the present disclosure, when the second object is highlighted, the sound of the second object is highlighted to match the second object displayed in the image.

[0174] Figure 5 This is an example diagram of resetting key frames shown in the embodiment of the present disclosure. Figure 5In it, the control indicated by L14 is the one for indicating reset. The user clicks on the node on the first-axis control indicated by L6, and the control indicated by L14 for indicating reset is displayed; when the user clicks on the control indicated by L14, the image frame indicated by the node of this click operation is reset and marked as not.

[0175] Figure 6 It is an example diagram of object focusing function switching shown in an embodiment of the present disclosure. Figure 6 In it, the user clicks on the control indicated by L4 to turn on the object focusing function. After the object focusing function is turned on, the control indicated by L4 is highlighted; the user clicks on the control indicated by L4 again to turn off the object focusing function. After the object focusing function is turned off, the control indicated by L4 cancels the highlight; among them, if the user turns off the object focusing, the original picture and the corresponding audio are restored.

[0176] Figure 7 It is a diagram of a media file processing device shown in an embodiment of the present disclosure. From Figure 7 it can be seen that it includes:

[0177] A first display module 701, configured to display a first image frame in the media file; wherein, the first image frame includes a first object and a second object, and the first object in the first image frame is highlighted compared to the second object;

[0178] A switching module 702, configured to, in response to detecting an object switching instruction, highlight the second object in the first image frame; wherein, the sound of the second object in the first image frame is prominent to match the highlighted second object.

[0179] In some embodiments, the switching module 702 is configured to display a first mark in the first image frame to highlight the second object; wherein, the first mark is located in the area where the second object is located.

[0180] In some embodiments, the device further includes:

[0181] A second display module, configured to display a second mark in the first image frame; wherein, the second mark is located in the area where the first object is located; the visual effect of the second mark is different from that of the first mark.

[0182] In some embodiments, the switching module 702 is configured to highlight the second object by adjusting the display effect of the area where the second object is located, and / or, adjusting the display effect of the area outside the second object.

[0183] In some embodiments, the switching module 702 is configured to highlight the second object by adjusting the pixel values of the region where the second object is located and / or the pixel values of the region outside the second object.

[0184] In some embodiments, the switching module 702 is configured to blur the pixel region outside the second object in the first image frame in response to detecting a blurring instruction.

[0185] In some embodiments, the apparatus further includes:

[0186] An adjustment module, configured to enhance the sound of the second object and / or eliminate or reduce the sound of the first object to highlight the sound of the second object.

[0187] In some embodiments, the apparatus further includes:

[0188] A marking module, configured to identify the first image frame affected by the object switching instruction as a key frame in response to detecting the object switching instruction;

[0189] The switching module 702 is configured to highlight the second object in each image frame between the key frame and the next key frame; wherein the sound of the second object in each image frame is prominent.

[0190] In some embodiments, the apparatus further includes:

[0191] A third display module, configured to display a control indicating reset in response to detecting an operation instruction for a first axis control with a key frame as a node;

[0192] A reset module, configured to reset the identification of the image frame indicated by the operation instruction for the first axis control as a non - key frame in response to detecting an operation instruction for the control indicating reset.

[0193] In some embodiments, the apparatus further includes:

[0194] A preview module, configured to preview a target image frame indicated by an operation instruction for a second axis control with the first image frame as a node in response to detecting the operation instruction;

[0195] The switching module 702 is configured to highlight the second object in the currently previewed target image frame; wherein the sound of the second object in the target image frame is prominent.

[0196] In some embodiments, the apparatus further includes:

[0197] A playback module, configured to play a second image frame before a switching object and a first image frame after the switching object that highlights a second object; wherein, the sound of the second object in the first image frame is prominent.

[0198] In some embodiments, the apparatus further includes:

[0199] A cancellation module, configured to cancel the highlighting of the second object in response to detecting a cancellation instruction; wherein, the prominent sound of the second object is cancelled.

[0200] In some embodiments, the apparatus further includes:

[0201] A first saving module, configured to save a second image frame before a switching object and a first image frame after the switching object that highlights the second object in response to detecting a first saving instruction; wherein, the sound of the second object in the first image frame is prominent.

[0202] In some embodiments, the apparatus further includes:

[0203] A second saving module, configured to save a second image frame before a switching object and an image of the second object in the first image frame after the switching object in response to detecting a second saving instruction.

[0204] Regarding the apparatus in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0205] Figure 8 It is a block diagram of an electronic device 800 shown in an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, a mobile computer, etc.

[0206] Refer to Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0207] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone call, data communication, camera operation, and recording operation. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0208] The memory 804 is configured to store various types of data to support the operation of the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0209] The power supply component 806 provides power to various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0210] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0211] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0212] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, and the peripheral interface module may be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0213] The sensor component 814 includes one or more sensors for providing an assessment of various aspects of the electronic device 800. For example, the sensor component 814 can detect the on / off state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor component 814 can also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. The sensor component 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0214] The communication component 816 is configured to facilitate communication between the electronic device 800 and other devices in a wired or wireless manner. The electronic device 800 can access a wireless network based on communication standards, such as Wi-Fi, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0215] In an exemplary embodiment, the electronic device 800 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0216] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the above instructions can be executed by a processor 820 of an electronic device 800 to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0217] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the foregoing audio-video editing method.

[0218] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0219] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A method for processing media files, characterized in that, The method includes: Displaying a first image frame in the media file; wherein, the first image frame includes a first object and a second object, and the first object in the first image frame is highlighted compared to the second object; In response to detecting an object switching instruction, highlighting the second object in the first image frame; wherein, the sound of the second object in the first image frame is prominent to match the highlighted second object.

2. The method according to claim 1, characterized in that, The highlighting the second object in the first image frame includes: Displaying a first mark in the first image frame to highlight the second object; wherein, the first mark is located in the area where the second object is located.

3. The method according to claim 2, characterized in that, The method further includes: Displaying a second mark in the first image frame; wherein, the second mark is located in the area where the first object is located; the visual effect of the second mark is different from that of the first mark.

4. The method according to claim 1 or 2, characterized in that, The highlighting the second object in the first image frame includes: By adjusting the display effect of the area where the second object is located, and / or, Adjusting the display effect of the area outside the second object to highlight the second object.

5. The method according to claim 4, characterized in that, The highlighting the second object by adjusting the display effect of the area where the second object is located, and / or adjusting the display effect of the area outside the second object includes: Highlighting the second object by adjusting the pixel values of the area where the second object is located, and / or the pixel values of the area outside the second object.

6. The method according to claim 5, characterized in that, The highlighting the second object by adjusting the pixel values of the area outside the second object includes: In response to detecting a blurring instruction, blurring the area outside the second object in the first image frame.

7. The method according to claim 1, characterized in that, The method further includes: Enhancing the sound of the second object, and / or, Eliminating or reducing the sound of the first object to highlight the sound of the second object.

8. The method according to claim 1, characterized in that, The method further includes: In response to detecting the object switching instruction, identifying the first image frame on which the object switching instruction acts as a key frame; The highlighting the second object in the first image frame includes: Highlighting the second object in each image frame between the key frame and the next key frame; wherein, the sound of the second object in each image frame is prominent.

9. The method according to claim 8, characterized in that, The method further includes: In response to detecting an operation instruction on a first axis control with a key frame as a node, displaying a control indicating reset; In response to detecting an operation instruction on the control indicating reset, resetting the identification of the image frame indicated by the operation instruction on the first axis control to a non-key frame.

10. The method according to claim 1, characterized in that, The method further includes: In response to detecting an operation instruction on a second axis control with the first image frame as a node, previewing the target image frame indicated by the operation instruction on the second axis control; The highlighting the second object in the first image frame includes: Highlighting the second object in the currently previewed target image frame; wherein, the sound of the second object in the target image frame is prominent.

11. The method according to claim 1, characterized in that, The method further includes: In response to detecting a play instruction, play the second image frame before the switching object and the first image frame after the switching object that highlights the second object; wherein, the sound of the second object in the first image frame is prominent.

12. The method according to claim 1, characterized in that, The method further includes: In response to detecting a cancellation instruction, cancel the highlighting of the second object; wherein, the prominent sound of the second object is cancelled.

13. The method according to claim 1, characterized in that, The method further includes: In response to detecting a first save instruction, save the second image frame before the switching object and the first image frame after the switching object that highlights the second object; wherein, the sound of the second object in the first image frame is prominent.

14. The method according to claim 1, characterized in that, The method further includes: In response to detecting a second save instruction, save the second image frame before the switching object and the image of the second object in the first image frame after the switching object.

15. A media file processing device, characterized in that, The apparatus includes: A first display module for displaying the first image frame in the media file; wherein, the first image frame includes a first object and a second object, and the first object in the first image frame is highlighted compared to the second object; A switching module for, in response to detecting an object switching instruction, highlighting the second object in the first image frame; wherein, the sound of the second object in the first image frame is prominent to match the highlighted second object.

16. An electronic device, characterized in that, Includes: A processor; A memory for storing processor-executable instructions; Wherein, the processor is configured to execute the method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 14.

Images